Streaming Music(SD, SPI ram, Network, etc. source)

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
Post Reply
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

I finally went after this idea years after the initial proof of concept in the Frog Feast time frame. I talked about this from dabbling with the idea, to the working solution, in nicksen782's Zelda thread. So that I don't totally saturate that thread with something that is meant to be a generic part of the kernel, I will talk more about this here. I wont repeat all that stuff I put there, but essentially this is a ram based circular buffer and song player modification that allows music to be streamed from anywhere. This is opposed to the standard flash based music, which 2-4K is common limiting the variety of music significantly in most Uzebox games.

It is generally working but I need to clean up the source to make it coexist nicely with the standard kernel. The streaming music can be used and configured by specifying:

Code: Select all

-DMUSIC_ENGINE=STREAM
-DSONG_BUFFER_SIZE=24##default if not specified
-DSONG_BUFFER_MIN=6##default if not specified
The hardest part is looping, because the buffering code is not aware of what the data means until it has already written ahead of the song player, the naive approach would end up with a buffer full of garbage after the last song bytes, then it is time to loop but we have no bytes to use. What is done, is to copy bytes after the loop start and put them after the loop end(minding to remove delta times that are normally skipped by that jump) so that as it reads past, it has the same data it should have so the whole thing an continue on without a hitch. That was the only real trick to it, and it requires a tool because it is a pain to do manually.

In addition to the buffer, it uses an added 16 bit value for loopEnd(it doesn't use songStart, so maybe we save those over stock) which is how the entire trick of it all works to avoid race conditions and loop correctly without ever reading bytes it can't use when it "reads past" the end of the song. I figured out how to properly handle out of data situations in a graceful way, so it no longer crashes, but simply extends the current delta time until enough data is available. This has the effect that if a buffer is too small and/or buffer filling operations are too slow for the particular part of the song, it just stretches out over a longer period of time. If it is too extreme it is noticeable but not horribly jarring, and better than other alternatives. So far I see promising results, where just 1 byte written to the buffer per frame with a 24 byte buffer, is able to play a reasonably complex song back, only slowing down gracefully during the most demanding parts. This seems key, since it is hard to predict individual SD card performance, individual game ticks, and developers are of course going to desire pushing things to the absolute limit instead of hauling a huge buffer around. I want to make performance maybe 50% better than this, so that such as 2 bytes read per frame on average, will not slow at all for most songs with a 16 byte buffer. This should work out, and also this would apply to flash music which could probably be 50+% smaller in this format. More to come on this.


BTW I would appreciate if anyone has opinions on uzeboxSoundEngine.c in the new "streaming-music" branch. I added an in process copy of the source that is being cleaned up, and also removed all things related to MIDI_IN. It just cleans the source up, and I do not think anyone ever did much with it, as well as the MIDI connector is not even standard since a long time ago. This seems wise to me because there are better ways to do it anyway(the existing implementation did not use HSYNC buffering, so missed bytes randomly anyway), but let me know if anyone disagrees.

I have been just feeding the stream from flash for now since it was more convenient, but this will give a rough idea of how to interface with it:

Code: Select all

	uint16_t soff = 1;
	while(!SongBufFull())
		SongBufWrite(pgm_read_byte(&Song1[soff++]));
	
	WaitVsync(60);
	StartSong(0);

	while(1){
		WaitVsync(1);	

		if(loopEnd){//we read past the end of the song..luckily it is padded with bytes from the loop start
			soff = (soff-loopEnd)+loopStart;
			loopEnd = 0;
		}

		for(uint8_t i=0;i<1;i++){
			if(SongBufFull())
				break;

			SongBufWrite(pgm_read_byte(&Song1[soff++]));
		}
	}
Let me know what you think. If anyone cares to mess with it now, attached is the only current song available in the correct format..not really a big whoop, it plays a song but can be thrown on an SD card to work on implementing it to a game. uzebox.h and uzeboxSoundEngine.c should be the only differences required over the stock kernel. I am working on the new compressed format, and have no plans of supporting this inefficient format however.
Attachments
song1.zip
(2.75 KiB) Downloaded 456 times
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

The compressed format needs to happen. The most common thing is a note on event, and I think there is a method to fit all this into about half the space. I only plan to support events for note on, program change, loop start, loop end, and song end. I think the rest can and should be done with patches, and are rarely or never used in an Uzebox game. Some events like loop start/etc. can be compressed to 1 byte down from 4+1 delta byte, others like note on that specify a channel, volume, and a note can be compressed to 2 down from 3+1 delta bytes.

What I mean by removing the delta byte is simply not to have a "time until next event" after every single event. Instead it is implied there is no delay between the next event. If that event turns out to be a "Tick End" event, then we know the following byte is a delta time until the next set of events. This is probably the biggest improvement which should equate to much higher performance and efficiency for the streaming music setup. About the byte stream, I am thinking something like this:

Code: Select all

1CCCVVVV, VNNNNNNN Note On: CCC = channel 0-4, VVVV V = volume>>3, NNNNNNN = note 0-127(versus 3 bytes+delta original)

0CCCPPPP, if PPPP == 15 then another byte PPPPPPPP Program Change: CCC = channel 0-4, PPPP or PPPP PPPPPPPP = patch num(save space if using patches 0-14)

0111EEEE,
	if EEEE == 0, Loop Start
	if EEEE == 1, Loop End
	if EEEE == 2, Song End
	if EEEE > 2, Tick End, DDDDDDDD = next delta time
	
ie.

01110000 //Loop Start
10001111 1111100//note on event,channel 0, max volume, note 60
10010000 0111101 //note on event, channel 0, lowest volume, note 61
01110011 0000100 // tick end event, next tick delay 7
etc.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by Artcfox »

D3thAdd3r wrote:The compressed format needs to happen. The most common thing is a note on event, and I think there is a method to fit all this into about half the space. I only plan to support events for note on, program change, loop start, loop end, and song end. I think the rest can and should be done with patches, and are rarely or never used in an Uzebox game. Some events like loop start/etc. can be compressed to 1 byte down from 4+1 delta byte, others like note on that specify a channel, volume, and a note can be compressed to 2 down from 3+1 delta bytes.
I would strongly argue for including Note Off events. Part of the reason I ported midiconv to C++ was so it could properly support Note Off events, and the difference in the song I composed between including the note off events and not including them was immense, the only downside was it used up so much flash space. What excites me the most about the potential to stream music from the SD card is that I can always include Note Off events, and it won't eat up any flash space, and it will always sound top notch.

Maybe this is just because the way I compose music is different from everyone else? Speaking of which, I really need to get my video tutorial out for composing music...
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Artcfox wrote:...the only downside was it used up so much flash space. What excites me the most about the potential to stream music from the SD card is that I can always include Note Off events, and it won't eat up any flash space, and it will always sound top notch.
Ah I remember some talk about that for Laser Puzzle. I was imagining that cutting features would somehow allow a more compact representation, but it really does not now that I had a chance to experiment some. Some things will be two bytes and some things one, and that is as good as it can be. The new delta design should save a lot, so might as well at least support the features the current MIDI player supports. If anyone knows of anything that might be nice, over the top of the existing support in the flash player, I am certainly open to it since it doesn't hurt someone trying to make a maximum performance song at all.

It seems that "Note Off" is implemented simply as a "Note On" with volume 0 so it should work in any design at least. Probably the other controllers are a bit less useful, but eh, why not.

Code: Select all

void TriggerNote(unsigned char channel,unsigned char patch,unsigned char note,unsigned char volume){
	Track* track=&tracks[channel];

	//allow only other music notes 
	if((track->flags&TRACK_FLAGS_PLAYING)==0 || (track->flags&TRACK_FLAGS_PRIORITY)==0){
			
		if(volume==0){ //note-off received

			
			//cut note if there's no envelope & no note hold
			if(track->envelopeStep==0 && !(track->flags&TRACK_FLAGS_HOLD_ENV)){
				track->noteVol=0;
			}

			track->flags&=(~TRACK_FLAGS_HOLD_ENV);//patchEnvelopeHold=false;
		}else{
		
			track->flags=0;//&=(~TRACK_FLAGS_PRIORITY);// priority=0;
			track->patchCommandStreamPos = NULL;
			TriggerCommon(track,patch,volume,note);
			track->flags|=TRACK_FLAGS_PLAYING;
		}

	}
}
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by Artcfox »

D3thAdd3r wrote:
Artcfox wrote:...the only downside was it used up so much flash space. What excites me the most about the potential to stream music from the SD card is that I can always include Note Off events, and it won't eat up any flash space, and it will always sound top notch.
Ah I remember some talk about that for Laser Puzzle. I was imagining that cutting features would somehow allow a more compact representation, but it really does not now that I had a chance to experiment some. Some things will be two bytes and some things one, and that is as good as it can be. The new delta design should save a lot, so might as well at least support the features the current MIDI player supports. If anyone knows of anything that might be nice, over the top of the existing support in the flash player, I am certainly open to it since it doesn't hurt someone trying to make a maximum performance song at all.

It seems that "Note Off" is implemented simply as a "Note On" with volume 0 so it should work in any design at least. Probably the other controllers are a bit less useful, but eh, why not.

Code: Select all

void TriggerNote(unsigned char channel,unsigned char patch,unsigned char note,unsigned char volume){
	Track* track=&tracks[channel];

	//allow only other music notes 
	if((track->flags&TRACK_FLAGS_PLAYING)==0 || (track->flags&TRACK_FLAGS_PRIORITY)==0){
			
		if(volume==0){ //note-off received

			
			//cut note if there's no envelope & no note hold
			if(track->envelopeStep==0 && !(track->flags&TRACK_FLAGS_HOLD_ENV)){
				track->noteVol=0;
			}

			track->flags&=(~TRACK_FLAGS_HOLD_ENV);//patchEnvelopeHold=false;
		}else{
		
			track->flags=0;//&=(~TRACK_FLAGS_PRIORITY);// priority=0;
			track->patchCommandStreamPos = NULL;
			TriggerCommon(track,patch,volume,note);
			track->flags|=TRACK_FLAGS_PLAYING;
		}

	}
}
Ah right, then note off commands should be supported automatically. :D
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

As was discussed in the thread for Matt's c++ rewrite of midiconv, the operations required to make the compressed format are not a good fit in the midiconv code base itself. At least I have no interest to have some modified version based on it floating around, just doesn't seem ideal . So I will make a stand alone command line tool that will perform the changes required to take standard midiconv output, and make them work for stream music. If anyone cares to comment this is the time. I am planning to not support at all the format that I am currently showing(the standard flash format more or less). Instead the player will only play compressed songs, but it will support all existing things the flash version does. I look at it like this, either way you are going to have to use a tool and/or manual massaging to get this stuff onto an SD card in a usable form. Any pros for supporting the current format?

Another tool that needs to exist to make this practical is something like hex2resource that can handle C array or binary files, starting at any offset and for any length, output to a resource file at any offset. Maybe something like "cat" on Linux could be coerced into something like this, but it seems better to have something totally catered to this without complex command line arguments to get it done. I guess I will work on these 2 things so I can just go the compressed route, otherwise I am in an awkward spot of developing something further with a format I hate because it is wasteful...maybe hate is a strong word. I think the flash player should stay the same format for legacy/compatibility, but when I go a route that takes away part of a ram tile, my eyes light up on how to make that smaller!!

Edit-Also is anyone opposed to removing the existing MIDI_IN stuff? I don't think it is worth the extra #define stuff for something I doubt more than 1 or 2 has every dabbled with and never talked about.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by Artcfox »

What do you mean by a resource file? Is thaf just a binary file, and you want to be able to convert Uzebox MIDI files from include files back into a binary form for your streaming player (accounting for sector size and looping bytes)?
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

Yes by resource file I mean just a raw file like "GAMENAME.DAT" that you put all the graphics, level, music, etc. data onto so your game rom can read it. Something like Alter Ego I just did this manually and that works, but it is a real hassle especially when you change designs, things get bigger and go over boundaries of other things...it quickly gets error prone and is a huge waste of the all too limited hobby time.

I think a dead simple program that does only this could be the ticket, where either it would get called multiple times with different arguments for the different files. Say the makefile generates 4 songs with midiconv, and the program gets called 4 times to put those files at known places. For multiple stuff like graphics, it would probably be ideal just to have a "script" and the makefile just triggers it specifying that. Probably no xml or anything like that, maybe just a comma seperated like:

Code: Select all

"input file, input type, input offset, input length, output file, output type, output offset(length implied by input length)"
"input file 2, input file 2 type,....."
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by Artcfox »

D3thAdd3r wrote:Yes by resource file I mean just a raw file like "GAMENAME.DAT" that you put all the graphics, level, music, etc. data onto so your game rom can read it. Something like Alter Ego I just did this manually and that works, but it is a real hassle especially when you change designs, things get bigger and go over boundaries of other things...it quickly gets error prone and is a huge waste of the all too limited hobby time.

I think a dead simple program that does only this could be the ticket, where either it would get called multiple times with different arguments for the different files. Say the makefile generates 4 songs with midiconv, and the program gets called 4 times to put those files at known places. For multiple stuff like graphics, it would probably be ideal just to have a "script" and the makefile just triggers it specifying that. Probably no xml or anything like that, maybe just a comma seperated like:

Code: Select all

"input file, input type, input offset, input length, output file, output type, output offset(length implied by input length)"
"input file 2, input file 2 type,....."
Gotcha. If you really want to be a masochist, you can probably crib a lot of what's needed from my Bugz level editor. I wrote it in C99 as a quick and dirty hack to another program I had already written for reading PNG files and generating include files for microcontroller stuff. I'm not sure if anyone has looked at how the level editor works, but that's how I was able to generate so many levels for Bugz so quickly, I was just drawing pixels on different layers using GIMP, and then I ran a custom script that converted all the .xcf files into pngs, and then it kicked off my level editor which chewed through all the pngs, and generated a whole bunch of #include files, including the metadata for the "table of contents" of the giant PROGMEM array that ultimately got included in the game. I can see the same concept being used for a resource file, so in the end rather than ending up with one giant C array in the source code, you end up with one binary resource file. It has a bunch of bit-packing routines and one easy to use function for using libpng that given a filename, will read that PNG file into memory (converting any of many PNG formats into 24-bit RGB).

It also has my custom C99 implementation of a red/black tree along with some extension functions for using it as a set, or for searching using a custom compare function. If you're really into algorithms, take a look inside the dirent_node.h file to see how I "inherited" from the red/black tree node to allow it to be fully customizable, without having to resort to messy pre-processor hacks, or void* pointers everywhere. It's way overkill for how I used it in the editor, but it's a really good demo of how to use my red/black tree implementation to store arbitrary data, and how to write extension functions for it (those extension functions live inside files with + in their file names) without having to modify any of the base library files.
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Streaming Music(SD, SPI ram, Network, etc. source)

Post by D3thAdd3r »

The Bugz level editor is quite nice, and certainly paid off in development. On this thing I had a bit of time to take a crack at it and came up with a pretty small solution which should do the trick, though it needs some finishing. It takes a C array generated by midiconv, and can output a compressed version(a format that I do not have an updated player for yet), to another C array file, or anywhere selected in a binary file. This should be enough to not require another program to do music, so it would just be another thing under midiconv in the makefile. It can output straight to stdout too in case that will be useful for someone.

For the other program that is just straight data(levels, graphics, etc.)I could see it being useful for the PNG stuff directly to a binary file, but then I wonder how much work it is to chop it up into tiles and maps. And also to implement the level of control on all that which might be required for different video modes(even Mode 1 likes might find storing lots of screen maps on the SD beneficial). So I am thinking more along the lines of a tool that is the opposite of bin2hex with perhaps a couple extra micro-features. That might be a little later, as I really want to get this streaming music stuff done while the iron is hot after talking about it all these years. I think I already have enough tool to do just that part at least.
Post Reply