Streaming Music

From Uzebox Wiki
Revision as of 01:42, 6 October 2017 by D3thAdd3r (talk | contribs)
Jump to: navigation, search

General Description

Streaming Music is a technique useful for playing music back from ram. This costs a small amount of ram for buffering, but enables the music to be stored in places other than the limited flash on the '644 as the stock kernel player does. Offloading this data to storage that has "virtually unlimited" storage like SD or "quite a bit" like SPI Ram can allow a great deal of music in a single game, without requiring extra flash space. This can help to finish a big game, where traditionally music can represent a large percentage of flash usage.

Streaming Music has the same general capabilities as the flash player, but internally it has many differences. To enable it requires a makefile flag:


The most obvious difference is the use of a circular buffer to store music data that will be played. The required size of this buffer might range from 8 bytes to perhaps 32 depending on the demand of the music, and the speed at which you can continually fill the buffer. The size of this buffer can be specified in the makefile like so:


You should probably start with a conservative value like this, and after your programs ram usage is set and the music done, roll this value back until it causes noticeable slow down of the music. It would also be best to test thoroughly during heavy CPU loads when there is a lot going on in the game. Increase slightly over that and you should have the most efficient use of ram, as well as being able to handle worst case scenarios that could arise without missing a beat. The required buffer size is directly related to how rapidly events will happen in the music. SDMusicDemo and SPIRamMusicDemo can be useful tools to detect stalls in the music with different buffer attributes.

Another item to consider is the speed at which you can fill the buffer, which is correlated to how large of a buffer you need. Basically if you can't be sure that you can always fill it fast(SD card), you might require more buffer. This is so that you are ahead of the curve with buffered data when a computationally expensive part of the game occurs, or in the case of SD card streaming, you cross a sector boundary. The Streaming Music system should never crash due to running out of data, but the music will "stretch" in time as gracefully as possible to fill the gaps until more data is there. This can range from not noticeable, to very harsh sounding depending on the music, where it happens, and how long the data drought is. The most likely culprit of this, if streaming from SD, would be the inter sector delay which could stop buffering for multiple frames while the card gets ready. Another likely problem, is that the user code simple does not finish game logic in time and frequently the buffering code starves for time.

Other Considerations

Because the song player relies on a buffer in this scheme, you will need to make provisions in your game to allow it to be filled every frame or at least every few frames. The recommended method is to use a customized version of WaitVsync(). The stock WaitVsync() does just what it should, it waits. Ideally we can use these wait cycles that would otherwise just be wasted, to do something useful like keeping the buffer full. In the ideal situation where game logic always finishes on time, this would have virtually no performance impact on the game. It can still work for games that sometimes miss a frame, although it then potentially aggravates the worst case scenario. Basically that would require more testing to make sure it never crashes, and perhaps some tweaking. An example of a custom WaitVsync() might look like this:

//Using SPI Ram:
void CustomWaitVsync(u8 frames){//we do a best effort to keep up to the demand of the song player.

		if(loopEnd){//we read past the end of the song..luckily it is padded with bytes from the loop start

			songOff = (songOff-loopEnd)+loopStart;
			loopEnd = 0;//since we immediately zero it so we don't keep doing it
			SpiRamSeqReadStart(0,(uint16_t)(songBase+songOff));//read from the start, plus the offset we "read past the end"

		while(!GetVsyncFlag()){//try to use cycles that we would normally waste

			if(doSongBuffer && !SongBufFull())


This tutorial will not repeat all the details found in the source code. Check out the demos "SDMusicDemo" and "SPIRamMusicDemo" in the "streaming-music" branch for full example implementations.

Data Preparation

The Streaming Music player does not use the same format as the standard flash player, nor the MOD player. It uses a binary compression scheme to represent the standard MIDI stream used in the flash player, in less space. The conversion results in smaller size for theoretically all songs, but in particular the large rapid songs receive the best compression. This is partially due to how the wait time between events is handled. The format was devised around that, as those songs are the most difficult to keep up with. In the extreme case this allows perhaps 40% less load on the ram buffer than the normal format would(the normal format is not supported at all), depending highly on the song. To use Streaming Music, you need music that is in this compressed format.

To get your music into the compressed format, you first start the same way you do for all MIDI music using the traditional "midiconv" to turn your MIDI file into a stream the flash player understands. So you can use the same method you are accustomed to for development, or the one described at Music Step By Step to create the data. It is perhaps even wise to first fine tune and finish a song using the flash player in a test rom, then do the further processing when you are done. The extra step is the use of "mconvert", which is a utility designed to take the direct output of "midiconv" and give you a compressed format directly usable for the Streaming Music player. At this point, you could store that output in flash and stream from there, and it would work(TODO I mean to add a direct compressed player, that does no buffering for flash data). The main appeal of this however, is likely the ability to offload music data entirely rather than the (sometimes small)compression/space savings it might offer in flash. That said, it might offer just enough savings to finish a flash only game without requiring the SD at all.

Once you have this data, there are potentially more steps required that depend on what you are trying to accomplish. Commonly it is expected the user will choose to stream the music from the SD card or the SPI Ram. In the case of the SPI Ram, the most likely place to store the data will be the SD card, and so the data preparation is the same, but there is the extra step in code to get this data from SD->SPI Ram. You need to take the MIDI stream output of "midiconv", and run it through "mconvert". Then finally take the output of "mconvert" and put it into an SD image(which your program might load into SPI ram) using "hex2bin", which can finally be buffered into the Streaming Music player for playback. The process is rather simple once done a couple times, and can be automated from the makefile. A good tutorial that might help in understanding the process can be found at SD Image Creation Tool. That will allow you to do everything from the command line or just a few extra steps in the makefile. Past this the "SDMusicDemo" and "SPIRamMusicDemo" should explain the programmatic details(though feel free to ask in the forums to improve the examples and this tutorial).

Image Format

To make things work, you will need to determine a data format to store your music on the SD card. The format you choose will directly relate to how your code will properly start a song. It is suggested to follow the SD Image Creation Tool tutorial example, and have a directory of pointers to each of the songs you will have. Then simply do a read from the directory to find the position you need to start buffering from, then call StartSong(). The custom WaitVsync() and the data stored at those offsets will do the rest to keep the whole thing running, and potentially drop seamlessly into your game without any issues.


Unfortunately the Streaming Music player is not as easy to understand as the standard flash player, and it requires more complexity in the user code than either flash MIDI or the MOD player to operate it. There is no way around this, but it is not too bad. StopSong() works the same, ResumeSong() also works the same, but StartSong() does not. Because the music data is implied to come from whatever is buffered, StartSong() takes no arguments in this version. Instead, the user code must ensure that the buffer has the beginning bytes of whatever song is to be played before StartSong() is called. Once this is done, the custom WaitVsync() takes care of all the details to keep the song running. This also implies that if you are currently playing a song, and wish to immediately play another song, you must purge the buffer of any data left over from the old song. This is not strictly necessary, as a small buffer may play out quickly on it's own and it might be acceptable to let that happen before it hits the start of the new song you are buffering. This is program specific and there are a lot of ways to pull it off. Again, the demos show 1 possible solution.

SD Card Latency

The current state of the SD Music Demo is such that it is not ideal for a fast action game, because it blocks program flow every time the SD card hits a 512 byte sector boundary. This is inherent in the way an SD card and simpleSD works, and currently no work around is provided that would allow for non-blocking operation. This might change in the future, and currently might be acceptable depending on your game type.

The much faster and more predictable SPI Ram version should be acceptable for any game, for which the reliance on SPI Ram is acceptable. Here we are basically using the SPI Ram as a fast cache for the SD card, so that we never have stalls. Currently I would recommend using the SPI Ram version until an acceptable state machine is created to minimize the effects of stalling for the SD version. In any case, the SPI Ram version will necessarily always have a higher performance no matter what, likely rivaling the flash music player in most cases. Of course, it will never reach the speed of the flash player with either SD or SPI ram, except where the usage of otherwise wasted cycles is enough to entirely offset it(for SPI Ram I believe this is common). Again, all the extra complication and steps is primarily intended to allow you to have "unlimited" music, and not use any extra flash for it. It is likely not useful in "small" games, but is believed to be very powerful for those willing to tackle extra complexity for loads of extra content.

Final Comments

I have attempted my very best to avoid making this whole subject seem mystical, because it definitely is not...perhaps arcane is a more accurate word :) There are a good number of details that were arbitrarily made, since that is the nature of an implementation such as this. So it will be greatly appreciated to get feedback in the forums on what doesn't make sense or is not covered in enough detail either here in this tutorial or the provided examples. For direct SD streaming, I think it might be possible without the stalls, but currently this is not a high priority. I base my priorities on the games I am currently making, and the games that others are currently making(ie. what gets discussed usually causes the most interest to do at the time), so don't feel shy about discussing unmet requirements there. In hopefully the near future there will be 2 full example games that use the SPI Ram method extensively, Adventures of Lolo and Alter Ego 2, though currently this is not implemented in either.