TriggerFx sometimes gets stuck on a note

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
User avatar
Artcfox
Posts: 955
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: TriggerFx sometimes gets stuck on a note

Post by Artcfox » Thu Jul 30, 2015 11:35 pm

uze6666 wrote:Woa, only 4 bytes free?!? You're really running after trouble! As a rule of thumb never go over 97-98% ram usage to leave some stack for the kernel interrupt. I can tell you that sound issues already happened to me a long time ago when I was pushing the ram a bit too tight (that and random crashes and glitches). I never profiled the kernel for its top stack usage but it's probably in the range 60-80 bytes. Usage is spread across:
  • Video rendering interrupt: pushes all registers and flags (33 bytes)
  • Video mode: uses the stack at varying levels, probably 10 bytes in average
  • Inline mixer: pushes 5 registers
  • Music engine: this one is in C and I tried to limit the number of function calls. But it could eat up to 50 bytes.
So yeah, if you triggerFX and the internal call stack is at it deepest (2 sub functions calls) and a rendering interrupt occurred at this precise time, I could well create such issue.
That's really good to know! Basically what I did before was keep adding RAM tiles until I got graphical glitches and then subtracted one RAM tile.

Sadly, it turns out the issue is only mostly solved. I play tested it for an hour with no glitches before I posted saying that I think it's fixed. After posting, the very next time I ran it, a note was sustained, but that's the only time it's ever happened, and I haven't been able to make it happen again.

Using this method to measure stack usage should include anything that the kernel does with the stack, since I presume if it's using more stack that it would overwrite the canary values with something else. The one time that I did get it to sustain a note, I looked and saw there were still 26 canary values untouched, so I guess there goes the theory of me being too close to the RAM limit and having something in the kernel blowing the stack.

User avatar
D3thAdd3r
Posts: 2424
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: TriggerFx sometimes gets stuck on a note

Post by D3thAdd3r » Fri Jul 31, 2015 12:21 am

As far as I can tell that is a good method to use, but it's not super obvious then why eliminating ram tiles should make it happen less but not stop entirely. For something that has went this far I'd step through every detail from the beginning, 1 more time than you think is necessary, just to make sure nothing was overlooked that would lead to a false understanding. Failing that, something else must be causing it(that is also aggravated by cycles or stack). If it comes down to it at the end of game development, some gross hack could easily be made to catch stuck notes/envelopes and simply make them decay.

I think it's probably just the envelopes and the patch has processed through entirely. You could do a quick check by printing the patch pointers out on the screen. Haven't tried this, but it should show rapidly fly through the addresses of the patch commands and end at the same point each time the effect was played. If you noticed the effect hanging coincided with that number getting stuck where it usually doesn't then you would know it was actually the patch command getting stuck. If that acted the same you would then know the patch is processing correctly and it's simply the bit of code:

Code: Select all

//uzeboxSoundEngine.c
	//
	// Process patches command streams & final volume
	//	
	for(unsigned char trackNo=0;trackNo<CHANNELS;trackNo++){
		track=&tracks[trackNo];

		//process patch command stream
		if((track->flags & TRACK_FLAGS_HOLD_ENV)==0){	//patchEnvelopeHold==false

			if(track->patchCommandStreamPos!=NULL && 
				track->patchCurrDeltaTime>=track->patchNextDeltaTime){			
//..................
You could try some assortment of these:

Code: Select all

PrintLong(5,1,tracks[2].patchCommandStreamPos);
PrintLong(5,2,tracks[2].patchCurrDeltaTime);
PrintLong(5,3,tracks[2].patchNextDeltaTime);
PrintLong(5,4,tracks[3].patchCommandStreamPos);
PrintLong(5,5,tracks[3].patchCurrDeltaTime);
PrintLong(5,6,tracks[3].patchNextDeltaTime);
At least you could tell that the patch was entirely processed.

User avatar
Artcfox
Posts: 955
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: TriggerFx sometimes gets stuck on a note

Post by Artcfox » Fri Jul 31, 2015 1:16 am

D3thAdd3r wrote:As far as I can tell that is a good method to use, but it's not super obvious then why eliminating ram tiles should make it happen less but not stop entirely. For something that has went this far I'd step through every detail from the beginning, 1 more time than you think is necessary, just to make sure nothing was overlooked that would lead to a false understanding. Failing that, something else must be causing it(that is also aggravated by cycles or stack). If it comes down to it at the end of game development, some gross hack could easily be made to catch stuck notes/envelopes and simply make them decay.

I think it's probably just the envelopes and the patch has processed through entirely. You could do a quick check by printing the patch pointers out on the screen. Haven't tried this, but it should show rapidly fly through the addresses of the patch commands and end at the same point each time the effect was played. If you noticed the effect hanging coincided with that number getting stuck where it usually doesn't then you would know it was actually the patch command getting stuck. If that acted the same you would then know the patch is processing correctly and it's simply the bit of code:
...
At least you could tell that the patch was entirely processed.
So it turns out that my debugging statement had a bug :oops: (I wasn't printing out actual numbers, just tiles and I tried to deduce the number based on the tile), and I shifted the high byte the wrong way, so it wasn't showing me the correct stack usage. I just added the digits 0-9 to my tileset, and wrote a function that can display digits:

Code: Select all

static void DisplayNumber(uint8_t x, uint8_t y, uint16_t n)
{
  uint8_t origx = x;
  do {
    SetTile(x--, y, (n % 10) + FIRST_DIGIT_TILE); // get next digit
  } while ((n /= 10) > 0);                        // delete it
  while (origx - x < 5)                           // pad with 0's up to 5 digits
    SetTile(x--, y, FIRST_DIGIT_TILE);
}
and with everything enabled (2 players, 6 monsters, 28 RAM tiles) it's showing 260 bytes free, dipping as low as 256 bytes free after playing it for a while, so I should be good on RAM.

I also printed out those patch statements, and for the most part the patchCommandStreamPos and patchNextDeltaTile show as 0. They sometimes briefly flash a number too fast for me to see, but both patchCurrDeltaTime values continually loop from 0-255, sometimes in sync with each other, and sometimes out of sync. I got it to trigger the glitch again, and they still loop over and over, no different from when it isn't glitching.

I then changed RAM_TILES to 32, which only leaves me with 16 bytes free, and after running it a while it dips lower and lower until it overflows the stack, at which point I got the sound glitch, and I noticed that the patchCommandStreamPos was briefly flashing a number that has 5 digits.

User avatar
D3thAdd3r
Posts: 2424
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: TriggerFx sometimes gets stuck on a note

Post by D3thAdd3r » Fri Jul 31, 2015 1:50 am

Artcfox wrote:I did find a way to reproduce it pretty regularly on actual hardware with a certain test level. The only thing that I need to do is to run right, hit jump (jump sound plays) and then stomp on an enemy (squished sound starts to play, and if I caught it at the right time, then it sustains a single note near the beginning of the squished sound), meanwhile I continue running to the right, and a few seconds later the note that's being sustained finishes, and the rest of the squished sound effect finishes playing, and things are quiet again.
You likely know the 5 digit number is the actual offset in the rom the kernel is reading from an interpreting as sound effect data. Each step in the effect, ie. 0,PC_PITCH,78,, of course takes 3 bytes. I had to reread to understand this problem again, and the above quote seems very important. The sound effects "stalls out" and does not continue on to the next step, and since a decay envelope is not set it "hangs" at a certain volume/note. The fact that it then continues and finishes the rest of the sound tells me it is absolutely the kernel not moving on to the next step(but why then does it do so later??). Even if the envelope value was being corrupted you could expect the patch to finish in the same amount of time. I wonder if adding some larger delays on each step of the sound effect could shed some light on this; being able to see the address and values longer.

If the TRACK_FLAGS_HOLD_ENV flag gets set and then later unset I would expect such a result. I wonder if there is some hidden out of bounds write happening like perhaps writing past vram. The canary values prove it's not the stack hitting it, so it seems necessary that something else is changing one or more of those relevant variables(deltatime,nextdeltatime,position,flags) discussed. I would grab a fresh copy of the kernel to be safe, since that code just works if left alone. I will have to check the kernel again, I may have forgotten how delta time works but it seems strange they continually run even when no sound effect is going.

Edit-Perhaps it's even warranted to question the compiler at this point?

User avatar
Artcfox
Posts: 955
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: TriggerFx sometimes gets stuck on a note

Post by Artcfox » Fri Jul 31, 2015 2:17 am

Yeah, it seems like something that's playing the note misses the end condition and it doesn't continue on until whatever it is incrementing/decrementing does a full wrap again and then it notices the end condition.

I slowed down the sound effects by changing the patches to hold the notes for 10 frames, and I can see tracks[2].patchCommandStreamPos goes to 14,xxx sometimes, but then it goes back to 0. I got it to sustain again, and it still went to 14,xxx and then back to 0 while the note was being sustained.

It would seem that both patchCommandStreamPos and patchNextDeltaTime are always 0, and the patchCurrDeltaTime continually increments until it wraps at 255, but if two sound effects are playing at once, then only tracks[2].patchCommandStreamPos briefly goes to 14,xxx and then back to 0. If the note is sustained and tracks[2].patchCommandStreamPos goes back to 0, jumping again (causing it to play a second sound) will cause it to go back to 14,xxx again.

User avatar
Artcfox
Posts: 955
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: TriggerFx sometimes gets stuck on a note

Post by Artcfox » Fri Jul 31, 2015 2:25 am

AHA!

I just got it to sustain the note again, and while it was being sustained, the tracks[2].patchCommandStreamPos got stuck on 14,162 for the entire time the note was being sustained and then just as it got unstuck it went back to 0.

Unfortunately I need to step away from the computer right now, but I'll check this thread as often as I can.

User avatar
D3thAdd3r
Posts: 2424
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: TriggerFx sometimes gets stuck on a note

Post by D3thAdd3r » Fri Jul 31, 2015 4:41 am

Ah my bad I always get the track numbers confused :oops: I think tracks[1] and tracks[2] should be the ones you are interested in. Sound effects will go on tracks[2] unless there is already another one playing it will resort to tracks[1], and tracks[3] probably never gets used with your current setup.

I'm assuming the byte of the rom at offset 14,162 point to some part of the command that gets stuck, otherwise that's really weird. Something is clobbering the kernels sound variable(s) for sure.

User avatar
Artcfox
Posts: 955
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: TriggerFx sometimes gets stuck on a note

Post by Artcfox » Sat Aug 01, 2015 6:22 pm

Okay, so I have some more data points. With the sound effects slowed down to 10 frames per note, and the debugging display changed so it's now printing tracks[1] and tracks[2], I can see both sound effects as they play through.

I got the note to stick three times, sometime on tracks[1] and sometime on tracks[2], and each time it sticks, the address 14172 for tracks[1 or 2].patchCommandStreamPos gets stuck while the note is hung. During this time, there are 292 bytes of untouched stack space, so it's unlikely to be a stack overflow issue.

Edit: Running it more, I found that sometimes it sticks without me being able to hear it at 14152.

User avatar
Artcfox
Posts: 955
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: TriggerFx sometimes gets stuck on a note

Post by Artcfox » Sat Aug 01, 2015 8:00 pm

So I started noticing that other variables near the beginning of my program were being overwritten randomly. Sometimes my level loader would catch that the level was invalid, other times it slipped by and loaded complete garbage (as if the currentLevel variable was out of bounds), other times the dead flag would be set on player 2, or player 2 would have the wrong starting coordinates.

As a test, I disabled the inline mixer in the Makefile

Code: Select all

-DSOUND_MIXER=0
and while some notes are still being sustained, at least my other variables aren't being clobbered. The plot thickens…

As an aside, without using the inline mixer, I have less RAM free, but the game runs fine (except for the TriggerFx glitches) with a "high water mark" of only 8 bytes of stack left untouched. ;)

Edit: Right now the only change I'm going to make is to re-enable the inline mixer, and I'll see what happens next.

Edit 2: I re-enabled the inline mixer, and no variables appear to be getting clobbered. I am confused.

Edit 3: The variables seem to eventually get clobbered regardless of using the inline mixer or not.

User avatar
uze6666
Site Admin
Posts: 4487
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: TriggerFx sometimes gets stuck on a note

Post by uze6666 » Sun Aug 02, 2015 12:23 am

Do you have array manipulation or pointer code that could overflow? Array out of bound could be a possibility for the garbage you saw. Since we are pretty much in the dark with this method of investigation, I implemented a controller capture mode in uzem. This allow to debug on uzem with GDB. Are you on Windows? If so:
To capture input:

Code: Select all

uzem -c YOURGAME.HEX
This will write to a file named yourgame.cap. Try replicate the sustained note issue and send me both your .hex and .cap files.

To replay a capture file:

Code: Select all

uzem -l YOURGAME.HEX
. The capture file with the same name (yourgame.cap) must be in the same folder.

Attached is uzem 1.31. an me playing ghosty ghost.

Code: Select all

uzem -l GGHOST.HEX
Attachments
uzem1_3_1_beta.zip
(488.35 KiB) Downloaded 67 times

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests