Mconvert tutorial: Difference between revisions

From Uzebox Wiki
Jump to navigation Jump to search
mNo edit summary
mNo edit summary
Line 1: Line 1:
==Introduction==
==Introduction==


TODO, PLEASE UNDERSTAND THIS IS NOT FINALIZED-------


mconvert is a utility designed specifically to accommodate data preparation for use with the streaming music player. It does not perform direct conversions of raw .MID files, as this is already accomplished well by midiconv. In general the goal is to let what midiconv does well in general, and focus on the specifics of streaming music that would be messy and a bad fit to place inside midiconv.
mconvert is a utility designed specifically to accommodate data preparation for use with the streaming music player. It does not perform direct conversions of raw .MID files, as this is already accomplished well by midiconv. The tools focuses only on the specifics of streaming music that would be messy and a bad fit to place inside midiconv.


mconvert will take the direct C array output provided by midiconv, and outputs data that is ready to be used in the streaming music player. This includes raw C arrays for direct inclusion in your game(costs flash but no buffer is required, not yet implemented), and abilities to create an SD file which contains all the music and an index to the start of individual song(the more common method). It is recommended to take a look at SPIRamMusicDemo to help understand this. There are several options available that will affect the format of the output data to your individual program needs.
mconvert will take the direct C array output provided by midiconv, and outputs data that is ready to be used in the streaming music player. This includes raw C arrays for direct inclusion in your game(costs flash but no buffer is required, not yet implemented), and abilities to create an SD file which contains all the music and an directory to the start of individual songs(the more common method). It is recommended to take a look at SPIRamMusicDemo to help understand this. There are several options available that will affect the format of the output data to your individual program needs.




Line 12: Line 11:




mconvert takes only one argument, and that is the name of a configuration file from which it shall operate. This configuration file is of a simple format which is easy to process, that allows the tool's source code to be small and simple to understand. The options should be enough for anyone's requirements. It is important to remember that, though the tool does cope with some deviance on spaces, etc. it is recommended to exactly follow the format described to avoid possible errors. The tool can be triggered from the command line like so:
mconvert takes only one argument, and that is the name of a configuration file from which it shall operate. This configuration file is of a simple format which is easy to process, that allows the tool's source code to be small and simple to understand. The options should be enough for anyone's requirements. It is important to remember that, though the tool does cope with some deviance on spaces, etc. it is recommended to closely follow the format examples. The tool can be triggered from the command line like so:
  mconvert ../path/to/config.file
  mconvert ../path/to/config.file


It is likely better to call it directly from your makefile, so that you do not need to manually intervene when things change. This is similar to what you will find in the tutorial for gconvert, in the makefile integration section: [[Generating Tiles and Maps with gconvert]]
It is likely better to call it directly from your makefile, so that you do not need to manually intervene when things change. This is similar to what you will find in the tutorial for gconvert, in the makefile integration section: [[Generating Tiles and Maps with gconvert]]


  ...
  ....
## Objects explicitly added by the user
LINKONLYOBJECTS =
## Include Directories
INCLUDES = -I"$(KERNEL_DIR)"
  ## Build
  ## Build
  all: SDMUSIC.DAT $(TARGET) $(GAME).hex $(GAME).eep $(GAME).lss $(GAME).uze size  
  all: SD_MUSIC.DAT $(TARGET) $(GAME).hex $(GAME).eep $(GAME).lss size
   
   
  ## Regenerate the music SD image
  ## Regenerate the music resources
  SDMUSIC.DAT: ../data/song1.inc ../data/song2.inc
  SD_MUSIC.DAT: ../data/sdImageConfig.cfg
  mconvert ../data/SDMusicImage.cfg
  $(UZEBIN_DIR)mconvert ../data/sdImageConfig.cfg
   
   
  ## Compile Kernel files
  ## Compile Kernel files
  uzeboxVideoEngineCore.o: $(KERNEL_DIR)/uzeboxVideoEngineCore.s
  ....
$(CC) $(INCLUDES) $(ASMFLAGS) -c  $<
 
..
Check out the SPIRamMusicDemo makefile in the streaming-music branch for a full example of this usage.
 




Line 41: Line 34:




The format of this file is not nearly as fancy or flexible as the xml found in gconvert. The source code is smaller, and uses nothing but the standard C library(and therefore easier to understand quickly) while doing everything that is required to get the job done: that is the reasoning behind this perhaps questionable design decision versus a full blown big xml setup. Lets look at an example of a configuration file:
The format of this file is not nearly as fancy or flexible as xml, but the entire program is 400% smaller source code than just the xml parser found in fancier tools. This was the reasoning for going this route, as it should flexible enough for all needs, and be easy to understand as it uses nothing external from the familiar C library. The following is an example of the format:
  1,512,1,MUSIC.DAT,
 
  ../tools/mconvert/song1.inc,0,
  1,0,512,1,MUSIC.DAT,
  ../tools/mconvert/song2.inc,3,
  ../tools/mconvert/song1.inc,0,NULL,
  ../tools/mconvert/song3.inc,0,
  ../tools/mconvert/song2.inc,3,NULL,
  ../tools/mconvert/song3.inc,0,NULL,
#this is a comment line


That is it. It should handle some minor changes from the ideal format, but it is wise to simple use this format to avoid potential issues. Do no put a newline after the last 0, in this example, to avoid what will appear like the start of a new entry to the simple parser. Let's think about what these arguments mean.
That is it. Let's think about what these arguments mean. The first line is the setup line which will dictate some things about how the entries will be processed. The first byte, '1' in this case, specifies to use the binary format as opposed to the C array hex format. This is expected to be the most common choice, and if you are doing SD/SPI Ram streaming you need this. 0 is for the flash version. The second number represents the start of the directory, which in this case is 0. The third number "512" represents where the first output byte of the first converted song will start at. 512 is a good location for most things. The next song will start immediately after the data for the first song. It is important to remember that for each converted song, mconvert will automatically fill in directory values starting the specified offset and moving up 4 bytes for each entry towards your music data. This allows songs to be anywhere in an image, and your code to easily be able to find them. 512 is an arbitrary choice that leaves lots of room, and has potential performance benefits in the case of SD streaming. The next item after this, '1' in this case, specifies whether the tool should apply padding to the end of songs. The padding, if enabled, will ensure that the total number of bytes output for each song is an even multiple of 512. These padding bytes will all be 0xFF(they should never be reached in a proper song, but if they were, this value would stop the song). The reason for padding out the data is primarily intended to help with SD streaming. SD cards have a penalty for crossing a 512 byte sector, and you also cannot start reading except directly on a sector boundary. By having all the music start on sector boundaries, the amount of wasted byte reads can be minimized which increases performance.


The first line is the setup line which will dictate some things about how the entries will be processed. The first byte, '1' in this case, specifies to use the binary format as opposed to the C array hex format. This is expected to be the most common choice, and if you are doing SD/SPI Ram streaming you need this. 0 is for the flash version(not yet implemented). The second number "512" represents where the first output byte of the first converted song will start at. 512 is a good location for most things. The next song will start immediately after the data for the first song. It is important to remember that for each converted song, mconvert will automatically fill in directory values starting from 0 and moving up 4 bytes for each entry towards your music data. This allows songs to be anywhere in an image, and your code to easily be able to find them. 512 is an arbitrary choice that leaves lots of room, and has potential performance benefits in the case of SD streaming. The next item after this, '1' in this case, specifies whether the tool should apply padding to the end of songs. The padding, if enabled, will ensure that the total number of bytes output for each song is an even multiple of 512. These padding bytes will all be 0xFF(they should never be reached in a proper song, but if they were, this value would stop the song). The reason for padding out the data is primarily intended to help with SD streaming. SD cards have a penalty for crossing a 512 byte sector, and you also cannot start reading except directly on a sector boundary. By having all the music start on sector boundaries, the amount of wasted byte reads can be minimized which increases performance.
Each line after that is just 3 arguments. The first argument is the path to the song file which you wish to convert. The order the song will be written in the directory is implied by the order you specify them in the configuration file. The second value, if a bit cryptic, is mainly aimed at advanced users...advanced MIDI users that are too lazy to delete spurious MIDI events from their source material ;) This is a decimal representation of a byte which is used as binary flags. These individual bits can be used to eliminate specific controller events from the stream, which can help improve performance and will likely not be missed. The flags can be OR'ed together in any configuration. The values are as so:


Each line after that is just 2 arguments. The first argument is the path to the song file which you wish to convert. The order the song will be written in the directory is implied by the order you specify them in the configuration file. The second value, if a bit cryptic, is mainly aimed at advanced users...advanced MIDI users that are too lazy to delete spurious MIDI events from their source material ;) This is a decimal representation of a byte which is used as binary flags. These individual bits can be used to eliminate specific controller events from the stream, which can help improve performance and will likely not be missed. The flags can be OR'ed together in any configuration. The values are as so:
  #define FILTER_CHANNEL_VOLUME 1//filter out channel volume events
  #define FILTER_CHANNEL_VOLUME 1//filter out channel volume events
  #define FILTER_EXPRESSION 2//filter out expression events
  #define FILTER_EXPRESSION 2//filter out expression events
Line 59: Line 53:


For those not familiar with bit flags I highly recommend taking the time to understand them, it's simple, but basically you can just add these flags together. If for instance you want to eliminate all channel volume events and also all tremolo volume events, you would specify 5 for the value here(1+4). If you want to eliminate spurious expression events(common with the NSF->MIDI method shown in [[Music Step By Step]]) as well as all tremolo rate events and note off events, specify the number 26 which is 2+8+16
For those not familiar with bit flags I highly recommend taking the time to understand them, it's simple, but basically you can just add these flags together. If for instance you want to eliminate all channel volume events and also all tremolo volume events, you would specify 5 for the value here(1+4). If you want to eliminate spurious expression events(common with the NSF->MIDI method shown in [[Music Step By Step]]) as well as all tremolo rate events and note off events, specify the number 26 which is 2+8+16
The final value is only useful if you specified C Array output by having a 0 for the first value on the first line. In that case, all output songs in the C array output file will be named accordingly to this argument. If doing binary, you can put anything here as it wont be used, but NULL makes things most clear.
There is more information in the Readme.txt file within the mconvert directory, if you are looking for more detailed information on any particular item. Feel free to ask on the forums as well of course!





Revision as of 02:40, 8 October 2017

Introduction

mconvert is a utility designed specifically to accommodate data preparation for use with the streaming music player. It does not perform direct conversions of raw .MID files, as this is already accomplished well by midiconv. The tools focuses only on the specifics of streaming music that would be messy and a bad fit to place inside midiconv.

mconvert will take the direct C array output provided by midiconv, and outputs data that is ready to be used in the streaming music player. This includes raw C arrays for direct inclusion in your game(costs flash but no buffer is required, not yet implemented), and abilities to create an SD file which contains all the music and an directory to the start of individual songs(the more common method). It is recommended to take a look at SPIRamMusicDemo to help understand this. There are several options available that will affect the format of the output data to your individual program needs.


Usage

mconvert takes only one argument, and that is the name of a configuration file from which it shall operate. This configuration file is of a simple format which is easy to process, that allows the tool's source code to be small and simple to understand. The options should be enough for anyone's requirements. It is important to remember that, though the tool does cope with some deviance on spaces, etc. it is recommended to closely follow the format examples. The tool can be triggered from the command line like so:

mconvert ../path/to/config.file

It is likely better to call it directly from your makefile, so that you do not need to manually intervene when things change. This is similar to what you will find in the tutorial for gconvert, in the makefile integration section: Generating Tiles and Maps with gconvert

....
## Build
all: SD_MUSIC.DAT $(TARGET) $(GAME).hex $(GAME).eep $(GAME).lss size

## Regenerate the music resources
SD_MUSIC.DAT: ../data/sdImageConfig.cfg
	$(UZEBIN_DIR)mconvert ../data/sdImageConfig.cfg

## Compile Kernel files
....

Check out the SPIRamMusicDemo makefile in the streaming-music branch for a full example of this usage.


Configuration File

The format of this file is not nearly as fancy or flexible as xml, but the entire program is 400% smaller source code than just the xml parser found in fancier tools. This was the reasoning for going this route, as it should flexible enough for all needs, and be easy to understand as it uses nothing external from the familiar C library. The following is an example of the format:

1,0,512,1,MUSIC.DAT,
../tools/mconvert/song1.inc,0,NULL,
../tools/mconvert/song2.inc,3,NULL,
../tools/mconvert/song3.inc,0,NULL,
#this is a comment line

That is it. Let's think about what these arguments mean. The first line is the setup line which will dictate some things about how the entries will be processed. The first byte, '1' in this case, specifies to use the binary format as opposed to the C array hex format. This is expected to be the most common choice, and if you are doing SD/SPI Ram streaming you need this. 0 is for the flash version. The second number represents the start of the directory, which in this case is 0. The third number "512" represents where the first output byte of the first converted song will start at. 512 is a good location for most things. The next song will start immediately after the data for the first song. It is important to remember that for each converted song, mconvert will automatically fill in directory values starting the specified offset and moving up 4 bytes for each entry towards your music data. This allows songs to be anywhere in an image, and your code to easily be able to find them. 512 is an arbitrary choice that leaves lots of room, and has potential performance benefits in the case of SD streaming. The next item after this, '1' in this case, specifies whether the tool should apply padding to the end of songs. The padding, if enabled, will ensure that the total number of bytes output for each song is an even multiple of 512. These padding bytes will all be 0xFF(they should never be reached in a proper song, but if they were, this value would stop the song). The reason for padding out the data is primarily intended to help with SD streaming. SD cards have a penalty for crossing a 512 byte sector, and you also cannot start reading except directly on a sector boundary. By having all the music start on sector boundaries, the amount of wasted byte reads can be minimized which increases performance.

Each line after that is just 3 arguments. The first argument is the path to the song file which you wish to convert. The order the song will be written in the directory is implied by the order you specify them in the configuration file. The second value, if a bit cryptic, is mainly aimed at advanced users...advanced MIDI users that are too lazy to delete spurious MIDI events from their source material ;) This is a decimal representation of a byte which is used as binary flags. These individual bits can be used to eliminate specific controller events from the stream, which can help improve performance and will likely not be missed. The flags can be OR'ed together in any configuration. The values are as so:

#define FILTER_CHANNEL_VOLUME	1//filter out channel volume events
#define FILTER_EXPRESSION	2//filter out expression events
#define FILTER_TREMOLO_VOLUME	4//filter out tremolo volume events
#define FILTER_TREMOLO_RATE	8//filter out tremolo rate events
#define FILTER_NOTE_OFF		16//filter out all NOTE_ONs which have a volume of 0(that is how Uzebox does "NOTE_OFFs")

For those not familiar with bit flags I highly recommend taking the time to understand them, it's simple, but basically you can just add these flags together. If for instance you want to eliminate all channel volume events and also all tremolo volume events, you would specify 5 for the value here(1+4). If you want to eliminate spurious expression events(common with the NSF->MIDI method shown in Music Step By Step) as well as all tremolo rate events and note off events, specify the number 26 which is 2+8+16

The final value is only useful if you specified C Array output by having a 0 for the first value on the first line. In that case, all output songs in the C array output file will be named accordingly to this argument. If doing binary, you can put anything here as it wont be used, but NULL makes things most clear.

There is more information in the Readme.txt file within the mconvert directory, if you are looking for more detailed information on any particular item. Feel free to ask on the forums as well of course!


Conclusion

This tool should be rather easy to use, and hopefully will do everything that is required to generate data which is usable in your game. If you need an extra feature, notice a bug, or just have a question, please feel free to address this in the streaming music thread on the forums. Thanks for reading, and I hope you enjoy the possibilities of limitless music in your game!

You may also be interested in the possibility of streaming PCM simultaneously as you stream music. If you can afford the ram in your game to include the vsync mixer, you can enjoy some massive resource offloading bliss! If that sounds interesting, check out Streaming PCM.