Difference between revisions of "Streaming Music"

From Uzebox Wiki
Jump to: navigation, search
(Created page with "==General Description== Streaming Music is a technique useful for playing music back from ram instead of flash as the standard player does. This cost a small amount of ram,...")
 
m
Line 23: Line 23:
  
  
Because the song player relies on a buffer in this scheme, you will need to make provisions in your game to allow it to be filled every frame or at least every few frames. The recommended method is to use a customized version of WaitVsync(). The stock WaitVsync() does just what it should, it waits. Ideally we can use these wait cycles that would otherwise just be wasted, to do something useful like keep the buffer full. In the ideal situation where game logic always finishes on time, this would have virtually no performance impact on the game. It can still work for games that sometimes miss a frame, although it then potentially aggravates the worst case scenario. Basically that would require more testing to make sure it never crashes. An example of a custom WaitVsync() might look like this:
+
Because the song player relies on a buffer in this scheme, you will need to make provisions in your game to allow it to be filled every frame or at least every few frames. The recommended method is to use a customized version of WaitVsync(). The stock WaitVsync() does just what it should, it waits. Ideally we can use these wait cycles that would otherwise just be wasted, to do something useful like keeping the buffer full. In the ideal situation where game logic always finishes on time, this would have virtually no performance impact on the game. It can still work for games that sometimes miss a frame, although it then potentially aggravates the worst case scenario. Basically that would require more testing to make sure it never crashes, and perhaps some tweaking. An example of a custom WaitVsync() might look like this:
  
 
  //Using SPI Ram:
 
  //Using SPI Ram:
Line 48: Line 48:
 
  }
 
  }
 
  }
 
  }
 +
 +
This tutorial will not repeat all the details found in the source code. Check out the demos "SDMusicDemo" and "SPIRamMusicDemo" in the "streaming-music" branch for full example implementations.
 +
 +
 +
 +
==Data Preparation==
 +
 +
 +
 +
The Streaming Music player does not use the same format as the standard flash player, nor the MOD player. It uses a binary compression scheme to represent the standard MIDI stream used in the flash player. This results in smaller size for theoretically all songs, but in particular the large rapid songs receive the best compression. The format was devised around that, as those songs are the most difficult to keep up with. In the extreme case this allows perhaps 40% less load on the ram buffer than the normal format would(the normal format is not supported at all), depending highly on the song. To use Streaming Music, you need music that is in this compressed format.
 +
 +
 +
To get your music into the compressed format, you first start the same way you do for all MIDI music using the traditional "midiconv" to turn your MIDI file into a stream the flash player understands. So you can use the same method you are accustomed to or the one described at [[Music Step By Step]] to create the data. It is perhaps even wise to first fine tune and finish a song using the flash player in a test rom, then do the further processing when you are done. The extra step is the use of "mconvert", which is a utility designed to take the direct output of "midiconv" and give you a compressed format directly usable for the Streaming Music player. At this point, you could store that output in flash and stream from there, and it would work. The main appeal of this however, is likely the ability to offload music data entirely rather than the (sometimes small)compression/space savings it might offer in flash. That said, it might offer just enough savings to finish a flash only game without requiring the SD at all.
 +
 +
 +
Once you have this data, there are potentially more steps required that depend on what you are trying to accomplish. Commonly it is expected the user will choose to stream the music from the SD card or the SPI Ram. In the case of the SPI Ram, the most likely place to store the data will be the SD card, and so the data preparation is the same, but there is the extra step in code to get this data from SD->SPI Ram. You need to take the MIDI stream output of "midiconv", and run it through "mconvert". Then finally take the output of "mconvert" and put it into an SD image(which your program might load into SPI ram), which can finally be buffered into the Streaming Music player for playback. The process is rather simple once done a couple times, and can be automated from the makefile. A good tutorial that might help in understanding the process can be found at [[SD Image Creation Tool]]. That will allow you to do everything from the command line or just a few extra steps in the makefile. Past this the "SDMusicDemo" and "SPIRamMusicDemo" should explain the programmatic details(though feel free to ask in the forums to improve the examples and this tutorial).
 +
 +
 +
 +
==Image Format==
 +
 +
 +
 +
To make things work, you will need to determine a data format to store your music on the SD card. The format you choose will directly relate to how your code will properly start a song. It is suggested to follow the [[SD Image Creation Tool]] tutorial example, and have a directory of pointers to each of the songs you will have. Then simply do a read from the directory to find the position you need to start buffering from, then call StartSong(). The custom WaitVsync() and the data stored at those offsets will do the rest to keep the whole thing running, and potentially drop seamlessly into your game without any issues.
 +
 +
 +
 +
==Playback==
 +
 +
 +
 +
Unfortunately the Streaming Music player is not as easy to understand as the standard flash player, and it requires more complexity in the user code than either flash MIDI or the MOD player to operate it. There is no way around this, but it is not too bad. StopSong() works the same, ResumeSong() also works the same, but StartSong() does not. Because the music data is implied to come from whatever is buffered, StartSong() takes no arguments in this version. Instead, the user code must ensure that the buffer has the beginning bytes of whatever song is to be played before StartSong() is called. Once this is done, the custom WaitVsync() takes care of all the details to keep the song running. This also implies that if you are currently playing a song, and wish to immediately play another song, you must purge the buffer of any data left over from the old song. This is not strictly necessary, as a small buffer may play out quickly on it's own and it might be acceptable to let that  happen before it hits the start of the new song you are buffering. This is program specific and there are a lot of ways to pull it off. Again, the demos show 1 possible solution.
 +
 +
 +
 +
 +
==SD Card Latency==
 +
 +
 +
 +
The current state of the SD Music Demo is such that it is not ideal for a fast action game, because it blocks program flow every time the SD card hits a 512 byte sector boundary. This is inherent in the way an SD card and simpleSD works, and currently no work around is provided that would allow for non-blocking operation. This might change in the future, and currently might be acceptable depending on your game type.
 +
 +
The much faster and more predictable SPI Ram version should be acceptable for any game, for which the reliance on SPI Ram is acceptable. Here we are basically using the SPI Ram as a fast cache for the SD card, so that we never have stalls. Currently I would recommend using the SPI Ram version until an acceptable state machine is created to minimize the effects of stalling for the SD version. In any case, the SPI Ram version will necessarily always have a higher performance no matter what, likely rivaling the flash music player in most cases. Of course, it will never reach the speed of the flash player with either SD or SPI ram, except where the usage of otherwise wasted cycles is enough to entirely offset it(for SPI Ram I believe this is common). Again, all the extra complication and steps is primarily intended to allow you to have "unlimited" music, and not use any extra flash for it. It is likely not useful in "small" games, but is believed to be very powerful for those willing to tackle extra complexity for loads of extra content.

Revision as of 22:47, 23 September 2017

General Description

Streaming Music is a technique useful for playing music back from ram instead of flash as the standard player does. This cost a small amount of ram, but enables the music to be stored in places other than the limited flash on the '644. Offloading this data to storage that has "virtually unlimited" storage like SD or "quite a bit" like SPI Ram can allow a great deal of music in a single game, without requiring extra flash space. This can help to finish a big game, where traditionally music can represent a large percentage of flash usage.

Streaming Music has the same general capabilities as the flash player, but internally it has many differences. To enable it requires a makefile flag:

-DMUSIC_ENGINE=STREAM

The most obvious difference is the use of a circular buffer to store music data that will be played. The required size of this buffer might range from 8 bytes to perhaps 32 depending on the demand of the music, and the speed at which you can continually fill the buffer. The size of this buffer can be specified in the makefile like so:

-DSONG_BUFFER_SIZE=24

You should probably start with a conservative value like this, and after your programs ram usage is set and the music done, roll this value back until it causes noticeable slow down of the music. It would also be best to test thoroughly during heavy CPU loads when there is a lot going on in the game. Increase slightly over that and you should have the most efficient use of ram, as well as being able to handle worst case scenarios that could arise without missing a beat. The required buffer size is directly related to how rapidly events will happen in the music.

Another item to consider is the speed at which you can fill the buffer, which is correlated to how large of a buffer you need. Basically if you can be guaranteed that you can fill it fast(SD card), you might require more buffer so that you are ahead of the curve with buffered data when a computationally expensive part of the game occurs. The Streaming Music system should never crash due to running out of data, but the music will "stretch" in time as gracefully as possible to fill the gaps until more data is there. This can range from not noticeable, to very harsh sounding depending on the music, where it happens, and how long the data drought is. The most likely culprit of this, if streaming from SD, would be the inter sector delay which could stop buffering for multiple frames while the card gets ready.


Other Considerations

Because the song player relies on a buffer in this scheme, you will need to make provisions in your game to allow it to be filled every frame or at least every few frames. The recommended method is to use a customized version of WaitVsync(). The stock WaitVsync() does just what it should, it waits. Ideally we can use these wait cycles that would otherwise just be wasted, to do something useful like keeping the buffer full. In the ideal situation where game logic always finishes on time, this would have virtually no performance impact on the game. It can still work for games that sometimes miss a frame, although it then potentially aggravates the worst case scenario. Basically that would require more testing to make sure it never crashes, and perhaps some tweaking. An example of a custom WaitVsync() might look like this:

//Using SPI Ram:
void CustomWaitVsync(u8 frames){//we do a best effort to keep up to the demand of the song player.

	while(frames){
		if(loopEnd){//we read past the end of the song..luckily it is padded with bytes from the loop start

			songOff = (songOff-loopEnd)+loopStart;
			loopEnd = 0;//since we immediately zero it so we don't keep doing it
			SpiRamSeqReadEnd();
			SpiRamSeqReadStart(0,(uint16_t)(songBase+songOff));//read from the start, plus the offset we "read past the end"
		}

		while(!GetVsyncFlag()){//try to use cycles that we would normally waste

			if(doSongBuffer && !SongBufFull())
				SongBufWrite(SpiRamSeqReadU8());						
		}

		ClearVsyncFlag();
		frames--;
	}
}

This tutorial will not repeat all the details found in the source code. Check out the demos "SDMusicDemo" and "SPIRamMusicDemo" in the "streaming-music" branch for full example implementations.


Data Preparation

The Streaming Music player does not use the same format as the standard flash player, nor the MOD player. It uses a binary compression scheme to represent the standard MIDI stream used in the flash player. This results in smaller size for theoretically all songs, but in particular the large rapid songs receive the best compression. The format was devised around that, as those songs are the most difficult to keep up with. In the extreme case this allows perhaps 40% less load on the ram buffer than the normal format would(the normal format is not supported at all), depending highly on the song. To use Streaming Music, you need music that is in this compressed format.


To get your music into the compressed format, you first start the same way you do for all MIDI music using the traditional "midiconv" to turn your MIDI file into a stream the flash player understands. So you can use the same method you are accustomed to or the one described at Music Step By Step to create the data. It is perhaps even wise to first fine tune and finish a song using the flash player in a test rom, then do the further processing when you are done. The extra step is the use of "mconvert", which is a utility designed to take the direct output of "midiconv" and give you a compressed format directly usable for the Streaming Music player. At this point, you could store that output in flash and stream from there, and it would work. The main appeal of this however, is likely the ability to offload music data entirely rather than the (sometimes small)compression/space savings it might offer in flash. That said, it might offer just enough savings to finish a flash only game without requiring the SD at all.


Once you have this data, there are potentially more steps required that depend on what you are trying to accomplish. Commonly it is expected the user will choose to stream the music from the SD card or the SPI Ram. In the case of the SPI Ram, the most likely place to store the data will be the SD card, and so the data preparation is the same, but there is the extra step in code to get this data from SD->SPI Ram. You need to take the MIDI stream output of "midiconv", and run it through "mconvert". Then finally take the output of "mconvert" and put it into an SD image(which your program might load into SPI ram), which can finally be buffered into the Streaming Music player for playback. The process is rather simple once done a couple times, and can be automated from the makefile. A good tutorial that might help in understanding the process can be found at SD Image Creation Tool. That will allow you to do everything from the command line or just a few extra steps in the makefile. Past this the "SDMusicDemo" and "SPIRamMusicDemo" should explain the programmatic details(though feel free to ask in the forums to improve the examples and this tutorial).


Image Format

To make things work, you will need to determine a data format to store your music on the SD card. The format you choose will directly relate to how your code will properly start a song. It is suggested to follow the SD Image Creation Tool tutorial example, and have a directory of pointers to each of the songs you will have. Then simply do a read from the directory to find the position you need to start buffering from, then call StartSong(). The custom WaitVsync() and the data stored at those offsets will do the rest to keep the whole thing running, and potentially drop seamlessly into your game without any issues.


Playback

Unfortunately the Streaming Music player is not as easy to understand as the standard flash player, and it requires more complexity in the user code than either flash MIDI or the MOD player to operate it. There is no way around this, but it is not too bad. StopSong() works the same, ResumeSong() also works the same, but StartSong() does not. Because the music data is implied to come from whatever is buffered, StartSong() takes no arguments in this version. Instead, the user code must ensure that the buffer has the beginning bytes of whatever song is to be played before StartSong() is called. Once this is done, the custom WaitVsync() takes care of all the details to keep the song running. This also implies that if you are currently playing a song, and wish to immediately play another song, you must purge the buffer of any data left over from the old song. This is not strictly necessary, as a small buffer may play out quickly on it's own and it might be acceptable to let that happen before it hits the start of the new song you are buffering. This is program specific and there are a lot of ways to pull it off. Again, the demos show 1 possible solution.



SD Card Latency

The current state of the SD Music Demo is such that it is not ideal for a fast action game, because it blocks program flow every time the SD card hits a 512 byte sector boundary. This is inherent in the way an SD card and simpleSD works, and currently no work around is provided that would allow for non-blocking operation. This might change in the future, and currently might be acceptable depending on your game type.

The much faster and more predictable SPI Ram version should be acceptable for any game, for which the reliance on SPI Ram is acceptable. Here we are basically using the SPI Ram as a fast cache for the SD card, so that we never have stalls. Currently I would recommend using the SPI Ram version until an acceptable state machine is created to minimize the effects of stalling for the SD version. In any case, the SPI Ram version will necessarily always have a higher performance no matter what, likely rivaling the flash music player in most cases. Of course, it will never reach the speed of the flash player with either SD or SPI ram, except where the usage of otherwise wasted cycles is enough to entirely offset it(for SPI Ram I believe this is common). Again, all the extra complication and steps is primarily intended to allow you to have "unlimited" music, and not use any extra flash for it. It is likely not useful in "small" games, but is believed to be very powerful for those willing to tackle extra complexity for loads of extra content.