Streaming Music

From Uzebox Wiki
Revision as of 05:49, 7 October 2017 by D3thAdd3r (talk | contribs)
Jump to: navigation, search

General Description

Streaming Music is a technique useful for playing music back from ram. This costs a small amount of ram for buffering, but enables the music to be stored in places other than the limited flash on the '644 as the stock kernel player does. Offloading this data to storage that has "virtually unlimited" storage like SD or "quite a bit" like SPI Ram can allow a great deal of music in a single game, without requiring extra flash space. This can help to finish a big game, where traditionally music can represent a large percentage of flash usage.

Streaming Music has the same general capabilities as the flash player, but internally it has many differences. To enable it requires a makefile flag:

-DMUSIC_ENGINE=STREAM

The most obvious difference is the use of a circular buffer to store music data that will be played. The required size of this buffer might range from 8 bytes to perhaps 32 depending on the demand of the music, and the speed at which you can continually fill the buffer. The size of this buffer can be specified in the makefile like so:

-DSONG_BUFFER_SIZE=24

You should probably start with a conservative value like this, and after your programs ram usage is set and the music done, roll this value back until it causes noticeable slow down of the music. It would also be best to test thoroughly during heavy CPU loads when there is a lot going on in the game. Increase slightly over that and you should have the most efficient use of ram, as well as being able to handle worst case scenarios that could arise without missing a beat. The required buffer size is directly related to how rapidly events will happen in the music. SDMusicDemo and SPIRamMusicDemo can be useful tools to detect stalls in the music with different buffer attributes.

Another item to consider is the speed at which you can fill the buffer, which is correlated to how large of a buffer you need. Basically if you can't be sure that you can always fill it fast(SD card), you might require more buffer. This is so that you are ahead of the curve with buffered data when a computationally expensive part of the game occurs, or in the case of SD card streaming, you cross a sector boundary. The Streaming Music system should never crash due to running out of data, but the music will "stretch" in time as gracefully as possible to fill the gaps until more data is there. This can range from not noticeable, to very harsh sounding depending on the music, where it happens, and how long the data drought is. The most likely culprit of this, if streaming from SD, would be the inter sector delay which could stop buffering for multiple frames while the card gets ready. Another likely problem, is that the user code simple does not finish game logic in time and frequently the buffering code starves for time.

You can also specify a minimum size that the song player will process, where if the buffered bytes are below this, it will stall for a frame. This adjusts the behavior where some things that should happen simultaneously, happen with a frame or more delay in between. This potentially stretches time a bit but can still sound good. The default behavior is likely fine for most things, but you can override this if you have a specific need(cannot buffer enough bytes for a very demanding part, and the time dilation is audible...unlikely). You really should not need to do this, and should investigate how to fill faster. The flag is:

-DSONG_BUFFER_MIN=2

To aid in debugging, you can enable a kernel variable called songStalls with:

-DSTREAM_MUSIC_DEBUG=1

This will keep track of every time that the song player stops because the buffered data is below SONG_BUFFER_MIN. This can be handy to test, as often times you cannot hear a stall at all because they are rapidly continued by further buffering. There are potentially other debugging things this can add, if there is user demand for it.


Other Considerations

Because the song player relies on a buffer in this scheme, you will need to make provisions in your game to allow it to be filled every frame or at least every few frames. The recommended method is to use a customized version of WaitVsync(). The stock WaitVsync() does just what it should, it waits. Ideally we can use these wait cycles that would otherwise just be wasted, to do something useful like keeping the buffer full. In the ideal situation where game logic always finishes on time, this would have virtually no performance impact on the game. It can still work for games that sometimes miss a frame, although it then potentially aggravates the worst case scenario. Basically that would require more testing to make sure it never crashes, and perhaps some tweaking. An example of a custom WaitVsync() might look like this:

//Using SPI Ram:
void CustomWaitVsync(u8 frames){//we do a best effort to keep up to the demand of the song player.

	while(frames){
		if(loopEnd){//we read past the end of the song..luckily it is padded with bytes from the loop start

			songOff = (songOff-loopEnd)+loopStart;
			loopEnd = 0;//since we immediately zero it so we don't keep doing it
			SpiRamSeqReadEnd();
			SpiRamSeqReadStart(0,(uint16_t)(songBase+songOff));//read from the start, plus the offset we "read past the end"
		}

		while(!GetVsyncFlag()){//try to use cycles that we would normally waste

			if(doSongBuffer && !SongBufFull())
				SongBufWrite(SpiRamSeqReadU8());						
		}

		ClearVsyncFlag();
		frames--;
	}
}

This tutorial will not repeat all the details found in the source code. Check out the demos "SDMusicDemo" and "SPIRamMusicDemo" in the "streaming-music" branch for full example implementations.


Data Preparation

The Streaming Music player does not use the same format as the standard flash player, nor the MOD player. It uses a binary compression scheme to represent the standard MIDI stream used in the flash player, in less space. The conversion results in smaller size for theoretically all songs, but in particular the large rapid songs receive the best compression. This is partially due to how the wait time between events is handled. The format was devised around that, as those songs are the most difficult to keep up with. In the extreme case this allows perhaps 40% less load on the ram buffer than the normal format would(the normal format is not supported at all), depending highly on the song. This does not include the extra loop data, which as a duplication of the beginning data does not increase the actual buffering load. Nor does this include the sector padding, if enabled in the conversion tool, which is meant to aid SD streaming(songs always start on a sector boundary). To use Streaming Music, you need music that is in this compressed format, and some attention to detail will be necessary to fit your specifics.


To get your music into the compressed format, you first start the same way you do for all MIDI music using the traditional "midiconv" to turn your MIDI file into a stream the flash player understands. So you can use the same method you are accustomed to for development, or the one described at Music Step By Step to create the data. It is perhaps even wise to first fine tune and finish a song using the flash player in a test rom, then do the further processing when you are done. The SPIRamMusicDemo should also make a good test bed for this. The extra step required here is the use of "mconvert", which is a utility designed to take the direct output of "midiconv" and give you a compressed format directly usable for the Streaming Music player. At this point, you could store that output in flash and stream from there, and it would work(TODO I mean to add a direct compressed player, that does no buffering for flash data). The main appeal of this however, is likely the ability to offload music data entirely rather than the sometimes modest compression/space savings it might offer in flash. That said, it might offer just enough savings to finish a flash only game without requiring the SD at all. For the SD image, depending on settings like sector padding, it is possible the total size is larger than the input. However, the number of actual bytes that must be buffered is always smaller than what the stock stream would have required, and in that case it is on the much larger SD instead of flash. There are some complexities there, but there is not necessarily a need to waste time understanding them as the tools are meant to spare you the details.


Once you have this data, there are potentially more steps required that depend on what you are trying to accomplish. Commonly it is expected the user will choose to stream the music from the SD card or the SPI Ram. In the case of the SPI Ram, the most likely place to store the data long term will be the SD card, and so the data preparation is the same, but there is the extra step in code to get this data from SD->SPI Ram. You need to take the MIDI stream output of "midiconv", and run it through "mconvert". It is a good idea to call mconvert directly from your makefile in a manner similar to how one uses gconvert. Some explanation for the use of this tool is necessary. A good tutorial that is intended to help in understanding the process can be found at mconvert tutorial. Past this the "SDMusicDemo" and "SPIRamMusicDemo" should explain the programmatic details(though feel free to ask in the forums to improve the examples and this tutorial).


Image Format

This has been simplified from the original inception, and mconvert makes more assumptions about the format because there are only a couple ways to reasonably do it. The tool will start outputting data at a location you specify in a configuration file, and it will keep a directory that moves upwards from bytes 0. Each entry in the directory is 4 bytes for a 32 bit variable. This allows songs to be placed anywhere reasonable. Because the user can specify where the song data starts, it is possible the directory data could overwrite it if care is not taken to ensure song data starts after all directory data. In most cases, it is advisable to start your song data at offset 512 for SPI Ram or SD streaming. This allows a very large number of songs safely, and has speed benefits for SD streaming especially. If you are using SPI Ram and you absolutely need a couple hundred extra bytes, you can start this closer to the index stack. It would be curious to do this for SD card streaming.


Playback

Unfortunately the Streaming Music player is not as easy to understand as the standard flash player, and it requires more complexity in the user code than either flash MIDI or the MOD player to operate it. There is no way around this, but it is not too bad. StopSong() works the same, ResumeSong() also works the same, but StartSong() does not. Because the music data is implied to come from whatever is buffered, StartSong() takes no arguments in this version. Instead, the user code must ensure that the buffer has the beginning bytes of whatever song is to be played before StartSong() is called. Once this is done, the custom WaitVsync() takes care of all the details to keep the song running. This also implies that if you are currently playing a song, and wish to immediately play another song, you must purge the buffer of any data left over from the old song. This is not strictly necessary, as a small buffer may play out quickly on it's own and it might be acceptable to let that happen before it hits the start of the new song you are buffering. This is program specific and there are a lot of ways to pull it off. Again, the demos show 1 possible solution.



SD Card Latency

The current state of the SD Music Demo is such that it is not ideal for a fast action game, because it blocks program flow every time the SD card hits a 512 byte sector boundary. This is inherent in the way an SD card and simpleSD works, and currently no work around is provided that would allow for non-blocking operation. This might change in the future, and currently might be acceptable depending on your game type.

The much faster and more predictable SPI Ram version should be acceptable for any game, for which the reliance on SPI Ram is acceptable. Here we are basically using the SPI Ram as a fast cache for the SD card, so that we never have stalls. Currently I would recommend using the SPI Ram version until an acceptable state machine is created to minimize the effects of stalling for the SD version. In any case, the SPI Ram version will necessarily always have a higher performance no matter what, likely rivaling the flash music player in most cases. Of course, it will never reach the speed of the flash player with either SD or SPI ram, except where the usage of otherwise wasted cycles is enough to entirely offset it(for SPI Ram I believe this is common). Again, all the extra complication and steps is primarily intended to allow you to have "unlimited" music, and not use any extra flash for it. It is likely not useful in "small" games, but is believed to be very powerful for those willing to tackle extra complexity for loads of extra content.


Final Comments

I have attempted my very best to avoid making this whole subject seem mystical, because it definitely is not...perhaps arcane is a more accurate word :) There are a good number of details that were arbitrarily made, since that is the nature of an implementation such as this. So it will be greatly appreciated to get feedback in the forums on what doesn't make sense or is not covered in enough detail either here in this tutorial or the provided examples. For direct SD streaming, I think it might be possible without the stalls, but currently this is not a high priority. I base my priorities on the games I am currently making, and the games that others are currently making(ie. what gets discussed usually causes the most interest to do at the time), so don't feel shy about discussing unmet requirements there. In hopefully the near future there will be 2 full example games that use the SPI Ram method extensively, Adventures of Lolo and Alter Ego 2, though currently this is not implemented in either.