Streaming Music(SD, SPI ram, Network, etc. source)

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by nicksen782 »

Artcfox wrote: Sun Apr 01, 2018 6:01 am And then define a custom TriggerFx function in your code that forces sound effects to only use channel 5, so the voice stealing algorithm doesn't interfere with the tracks playing your music:
So, this would allow the 4 channels to work with streaming music from SPIRAM and a sound effect from FLASH?

What if I could store them both in FLASH but with the new more condensed data format?
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by Artcfox »

nicksen782 wrote: Fri Apr 06, 2018 3:56 am
Artcfox wrote: Sun Apr 01, 2018 6:01 am And then define a custom TriggerFx function in your code that forces sound effects to only use channel 5, so the voice stealing algorithm doesn't interfere with the tracks playing your music:
So, this would allow the 4 channels to work with streaming music from SPIRAM and a sound effect from FLASH?

What if I could store them both in FLASH but with the new more condensed data format?
You'll have to ask D3thAdd3r about that; I haven't messed with the SPIRAM stuff at all.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Yes, you can do all the same things for sound effects whether streaming(buffered or bufferless submodes) or the stock MIDI player. This is because the channels and patch commands operate at a lower level than any music player. Some confusion could be had by the SPIRamMusicDemo, in that it appears to use lots of channels for music while allowing 4 PCM sound effects, which would be possible with the flash/bufferless version also, or the stock MIDI player. There were no changes to channel/mixer code to make streaming music work, and the demo can do so only because it is mixing the PCM data at a lower level than channels/patch commands, with direct modification of the sound buffer.

Toorum's Quest, Block Boy, and some others show a custom method to trigger sound effects on the 5th channel. The music could also use the 5th channel, but it doesn't, so in that sense it is dedicated to sound effects by design. Alec explained the concept somewhere in the forums and I have been using it ever since, as it is quite nice for many things.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

@nicksen782 have you tried the new repository here? Looking at this more, I am quite sure that the noise issues are unrelated to streaming music, as the older version of Alter Ego based off a prior main branch never had the issue. It really affects Alter Ego, and I don't even want to try and modify the effects to work with the new caveats. So perhaps I can try to put some work towards it and see if I couldn't at least make some testing which can help. In my mind, it is not impossible to know exactly how it will sound just by the code, as you could output the values to a table and compare it against the old version for all possible noise values.

I created a pull request and they should be able to be automatically merged. I would appreciate if anyone will take a look at that.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by nicksen782 »

Your repo had the latest from Uzebox master as of 9 days ago, right?

Initially I could not compile. I got this error:

Code: Select all

../src/code/bb_music.c: In function ‘N782_changeSong’:
../src/code/bb_music.c:136:3:  ERROR:  too few arguments to function ‘StartSong’
   StartSong();
   ^
In file included from ../src/code/BubbleBobble.h:8:0,
                 from ../src/code/BubbleBobble.c:108:
../kernel/uzebox/kernel/uzebox.h:110:16: note: declared here
    extern void StartSong(const char *song);
Odd. I was pretty certain that I had the streaming music done solid. No crashes, buffer weirdness. I had a good system. Something in the kernel must have changed. I found this in uzeboxSoundEngine.c "#if STREAM_MUSIC_RAM == 1". I found another one of this in uzebox.h "#if STREAM_MUSIC_RAM". New compile flag! I don't remember this being required before.

This is how the music part of my Makefile looks now:

Code: Select all

# Music options
KERNEL_OPTIONS += -DMUSIC_ENGINE=STREAM -DSTREAM_MUSIC_DEBUG=1 -DSONG_BUFFER_SIZE=16
KERNEL_OPTIONS += -DSTREAM_MUSIC_RAM=1
KERNEL_OPTIONS += -DSOUND_CHANNEL_5_ENABLE=0 -DSOUND_MIXER=1 -DSONG_SPEED=1
Fortunately, the last thing I finished in Bubble Bobble was the music test screen. I confirmed that it worked. I also played through the game and nothing unexpectedly broke along the way.

As far as I can tell, this works.

Are my Makefile compile flags correct?

If STREAM_MUSIC_RAM is set is that just for songs? What about sound effects?
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Yes I only mentioned that change once somewhere lost in this thread, and I updated the streaming music wiki page at that time. You figured it out correctly, by default just enabling streaming will use the flash version which takes the same arguments as stock MIDI. With the ram flag you added, it is the buffered version you are looking for which previously was default (not requiring the extra flag), as the flash version did not exist initially.

Streaming music has no bearing on sound effects. There was talk about streaming sound effects, but none of that is implemented in the kernel at this time. If it was implemented in the future, it would likely be a separate thing below the music player anyway. So basically, all things true for sound effects while using MIDI, are also true for either form of streaming music. Let me know if you have any issues.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by nicksen782 »

D3thAdd3r wrote: Mon Apr 09, 2018 4:02 am ...by default just enabling streaming will use the flash version which takes the same arguments as stock MIDI.
So, to use the newer compressed midi but with flash, I would then NOT include the "STREAM_MUSIC_RAM=1" and instead of including the binary sound into a data file loaded into SPIRAM, I would just use a normal include in the game code like before? But with the benefits of the smaller FLASH footprint for each song?
D3thAdd3r wrote: Mon Apr 09, 2018 4:02 am With the ram flag you added, it is the buffered version you are looking for which previously was default (not requiring the extra flag)
The buffering is now optional then? I can chose not to buffer meaning whatever I want to play would need to be in FLASH. Right? This implies that you can chose between the original stock music player or the new streaming one (even if that stream is coming from FLASH and not SPIRAM/SDCARD?

Just to be certain then: "MUSIC_ENGINE=STREAM" activates the new music player. For FLASH only I would not need "SONG_BUFFER_SIZE=16".
Furthermore, "STREAM_MUSIC_DEBUG=1" only works with "songStalls" which is optional, right?
D3thAdd3r wrote: Mon Apr 09, 2018 4:02 am So basically, all things true for sound effects while using MIDI, are also true for either form of streaming music. Let me know if you have any issues.
"TriggerFx(19,0xff,true);" would play a sound effect (whatever 19 happens to be.) You would still need to do "InitMusicPlayer(patches);" beforehand, right? Are you saying that the new streaming music has no bearing on sound effects? Sound effects are patches consisting of some notes? What is the difference between that and a song other than the length?
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

nicksen782 wrote: Mon Apr 09, 2018 4:17 am So, to use the newer compressed midi but with flash, I would then NOT include the "STREAM_MUSIC_RAM=1" and instead of including the binary sound into a data file loaded into SPIRAM, I would just use a normal include in the game code like before? But with the benefits of the smaller FLASH footprint for each song?
Correct. It should save a good deal of space, and also a couple bytes of ram over the MIDI player due to some optimizations. It is called the same way from code, with StartSong(songName), etc. None of the juggling required like for the buffered version.
nicksen782 wrote: Mon Apr 09, 2018 4:17 am The buffering is now optional then? I can chose not to buffer meaning whatever I want to play would need to be in FLASH. Right? This implies that you can chose between the original stock music player or the new streaming one (even if that stream is coming from FLASH and not SPIRAM/SDCARD?
Yes, streaming music has 2 modes now and basically this will be the final state as I see it. "-DMUSIC_ENGINE=STREAMING" is for FLASH bufferless operation, and "-DMUSIC_ENGINE=STREAMING -DSTREAM_MUSIC_RAM=1 -DSONG_BUFFER_SIZE=XX" is for ram buffered version. The caveats of all the buffered stuff you already know, and none of that part has changed in a good while so should work just like before. The recent changes were just to add the flash version, which basically just replaces the circular buffer code with a pointer version similar to the stock MIDI player.
nicksen782 wrote: Mon Apr 09, 2018 4:17 am Just to be certain then: "MUSIC_ENGINE=STREAM" activates the new music player. For FLASH only I would not need "SONG_BUFFER_SIZE=16".
Furthermore, "STREAM_MUSIC_DEBUG=1" only works with "songStalls" which is optional, right?
Yes, no buffer should be specified for the FLASH only version as it does not use a buffer, rather it reads from a pointer that always moves forward until it reaches a song end, or a loop end, just like the MIDI player but in a different format. songStalls is the only thing STREAM_MUSIC_DEBUG does, and it's only use might be for testing how small a buffer size you can get away with. There would be no real purpose to leave it enabled in a game release.
nicksen782 wrote: Mon Apr 09, 2018 4:17 am "TriggerFx(19,0xff,true);" would play a sound effect (whatever 19 happens to be.) You would still need to do "InitMusicPlayer(patches);" beforehand, right? Are you saying that the new streaming music has no bearing on sound effects?
Yes.
nicksen782 wrote: Mon Apr 09, 2018 4:17 am What is the difference between that and a song other than the length?
A sound effect and a song are different ways of controlling patches, but ultimately either one of those relies on the channel playing some sort of patch. You could well make a short song inside a single patch with appropriate timings between events, but this would not be space efficient to repeat envelopes, etc for every note, nor fun to develop. It would also get messy trying to control more than one channel, as sound effects are not optimized for this, though several games do have multi-channel sound effects. You could I suppose also start a song that triggered a specific patch, in effect using StartSong(x) to trigger a sound effect in an inefficient way. The point is, in general there are things you want from a sound effect(don't clobber the music, have some basic priority system among sound effects), and things you want from a music engine(assume the patch should always play ignoring what sound effects are doing, and also pass a specific note in addition to the volume). By making these assumptions, both can be done more optimally for the types of things they are likely to be used for. But in some sense yes, they are just different ways of controlling the channels underneath, and at the end of the line it is just patches. It is just safe to assume that most of the time a song will consist of a linear series of patch triggers, where sound effects are generally not as predictable and often based on game logic dictated by human input. Thus, the assumptions you can make about either one of them is quite different thus the different implementations.

Easiest is to look how TriggerFx() works:

Code: Select all

void TriggerFx(unsigned char patch,unsigned char volume,bool retrig){
	unsigned char channel;
	
	unsigned char type=(unsigned char)pgm_read_byte(&(patchPointers[patch].type));

	//find the channel to play the fx
	//try to steal voice 2 then 1
	//never steal voice 0, reserve it for lead melodies
	if(type==1 || (type==2 && MIXER_CHAN4_TYPE == 1)){
		//noise or PCM channel fx		
		channel=3;
	}else if(type==2){
		channel=4;
	}else if( (tracks[1].flags&TRACK_FLAGS_PRIORITY)==0 || (tracks[1].fxPatchNo==patch && (tracks[1].flags&TRACK_FLAGS_PRIORITY)!=0 && retrig==true)){ //fx already playing
		channel=1;
	}else if( (tracks[2].flags&TRACK_FLAGS_PRIORITY)==0 || (tracks[2].fxPatchNo==patch && (tracks[2].flags&TRACK_FLAGS_PRIORITY)!=0 && retrig==true)){ //fx already playing				
		channel=2;
	}else{
		//both channels have fx playing, use the oldest one
		if(tracks[1].patchPlayingTime>tracks[2].patchPlayingTime){
			channel=1;
		}else{
			channel=2;
		}
	}				

	Track* track=&tracks[channel];
	track->flags=TRACK_FLAGS_PRIORITY; //priority=1;
	track->patchCommandStreamPos = NULL;
	TriggerCommon(track,patch,volume,80);//<<<<======================
	track->flags|=TRACK_FLAGS_PLAYING;
}
There is some logic to cooperate with the music, and hopefully cooperate well with other sound effects as well. I have often times done a different TriggerFx, which takes more types of priority into account and in some games I would recommend this. Either way, the code will be nothing like a song player.

Then look at a snippet of the stock MIDI player that handles note on events:

Code: Select all

...
					switch(lastStatus&0xf0){

						//note-on
						case 0x90:
							//c1 = note						
							c2=pgm_read_byte(songPos++)<<1; //get volume
						
							if(tracks[channel].flags|TRACK_FLAGS_ALLOCATED){ //allocated==true
								TriggerNote(channel,tracks[channel].patchNo,c1,c2);//<========================
							}
break;
...
This is a snippet from the streaming player:

Code: Select all

...
				}else if((c1 & 0b11100000) < 0b10100000){//0bCCCXXXXX, if (CCC>>5) is 0-4, then it is a Note On
					//Note On: 0bCCCVVVVV, VNNNNNNN = CCC = channel, VVVVV V = volume, NNNNNNN = note
					channel = (c1>>5);//get channel
					c1 = (c1 & 0b00011111);//get volume 5 LSBits
					c2 = SongBufRead();//get packed note and MSBit of volume
					c1 |= (c2 & 0b10000000)>>2;//add 1 MSBit to previous LSBits for 6 bits total volume
					c1 <<= 2;//convert our 6 bits to: 7 bit converted to 8 bit volume of original format
					c2 &= 0b01111111;//mask out the MSbit of volume, leaving just the note
					//c2 = note, c1 = volume
					if(tracks[channel].flags|TRACK_FLAGS_ALLOCATED)//allocated==true
						TriggerNote(channel,tracks[channel].patchNo,c2,c1);//<==============================
...
They both use a totally different format to encode song data, but once it is decoded, it is sent as a raw command for the channels below using TriggerNote(). This will cause a channel to play a patch, just like a sound effect. However the default TriggerFx() does not allow you to specify a specific note(and indeed sound effects usually set PC_PITCH manually), where of course this isn't ideal for most music.

You can see TriggerFx() is something like a wrapper over TriggerCommon(), tailored to control the channels below in a useful manner for sound effects. TriggerCommon() itself, is the common code between what is useful for controlling channels for sound effects as well as music notes; they are only different things conceptually based on how you use them. Looking at TriggerNote() which the music players use:

Code: Select all

void TriggerNote(unsigned char channel,unsigned char patch,unsigned char note,unsigned char volume){
	Track* track=&tracks[channel];

	//allow only other music notes 
	if((track->flags&TRACK_FLAGS_PLAYING)==0 || (track->flags&TRACK_FLAGS_PRIORITY)==0){
			
		if(volume==0){ //note-off received

			
			//cut note if there's no envelope & no note hold
			if(track->envelopeStep==0 && !(track->flags&TRACK_FLAGS_HOLD_ENV)){
				track->noteVol=0;
			}

			track->flags&=(~TRACK_FLAGS_HOLD_ENV);//patchEnvelopeHold=false;
		}else{
		
			track->flags=0;//&=(~TRACK_FLAGS_PRIORITY);// priority=0;
			track->patchCommandStreamPos = NULL;
			TriggerCommon(track,patch,volume,note);//<=============================
			track->flags|=TRACK_FLAGS_PLAYING;
		}

	}
}
We see that TriggerNote() is a similar wrapper as TriggerFx() over the top of TriggerCommon(), geared more for things that are useful for music notes. A channel just does whatever it was last instructed to do. It does know what set it to that state, nor does it know what music player you are using if any, it follows the patch whether it would be considered a sound effect or an instrument patch. So channels are basically the part that does the actual work of sound generation, where song players or TriggerFx() are something like management in a sense. You could manually control the channels to generate sound effects, or music, but you would be manually trigger patches for these purposes. The only difference would be if you used the vsync mixer, where you can write directly to a sound buffer. At that point, you could even get rid of the channels themselves and just do whatever you wanted. For the vast majority of cases that simply would be a waste of development time of course, so these general purpose solutions are quite powerful.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by Artcfox »

Great walkthrough of what it does! And just so I'm clear, this is the pull request in question, right?

https://github.com/Uzebox/uzebox/pull/92
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

That is it, and to the best of my knowledge it is a working merge of master and streaming music. More or less the final state of it.
Post Reply