mconvert - MIDI Compressed Format Tool

Topics on software tools like TileStudio, comments on documentation and tutorials (or the lack of) should go here.
User avatar
Artcfox
Posts: 944
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by Artcfox » Wed Jul 05, 2017 6:47 am

I haven't tested it, but it sounds like it would be good.

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Sat Oct 07, 2017 6:26 pm

Some progress on this, I am going to do further testing to make sure there are no bugs. Things are much different from the previous version, and you can no longer specify anything on the command line except the name of a file that contains the configuration data. The format is very simple, and the entire code is currently ~400 lines and only uses the standard C library.

This is an example of the usage now:

Code: Select all

mconvert ../data/sdImageConfig.cfg
Below is an example of my test configuration, which converts all the songs in Alter Ego to the compressed format and includes them in an SD image. I feel this format is good for all uses

Code: Select all

1,0,512,1,SD_MUSIC.DAT,
#this is a comment line, the above line specifies: binary output, directory starting at 0, songs starting at offset 512, pad song data to multiples of 512, to output file SD_MUSIC.DAT
../data/stage1song.inc,0,NULL,
#the above line specifies: song file name, no controller filters(explained in readme and tutorial), NULL array name(since not C output)
../data/stage2song.inc,0,NULL,
../data/stage3song.inc,0,NULL,
../data/stage4song.inc,0,NULL,
../data/stage5song.inc,0,NULL,
#this is a comment line ../data/someSongNotNeeded.inc,0
../data/titlesong.inc,0,NULL,
../data/welldonesong.inc,0,NULL,
This is an example of the output, which I feel displays all pertinent information:

Code: Select all

root@lab-dell:~/uzebox.git/branches/streaming-music/demos/SPIRamMusicDemo/default# ../../../bin/mconvert ../data/sdImageConfig.cfg

	Binary output, directory at 0, data at  512, to SD_MUSIC.txt:
	+= ../data/stage1song.inc,filter:0,offset:512
		Run time: 48 seconds
		Input data size: 3472
		Output data size: 2919(+256 loop +409 pad)
		Average bytes per frame: 1.102048
		Average bytes per second: 66.122874

	+= ../data/stage2song.inc,filter:0,offset:4096
		Run time: 43 seconds
		Input data size: 3014
		Output data size: 2207(+256 loop +97 pad)
		Average bytes per frame: 0.950231
		Average bytes per second: 57.013889

	+= ../data/stage3song.inc,filter:0,offset:6656
		Run time: 38 seconds
		Input data size: 2151
		Output data size: 1626(+256 loop +166 pad)
		Average bytes per frame: 0.825077
		Average bytes per second: 49.504603

	+= ../data/stage4song.inc,filter:0,offset:8704
		Run time: 31 seconds
		Input data size: 3471
		Output data size: 2804(+256 loop +12 pad)
		Average bytes per frame: 1.604615
		Average bytes per second: 96.276875

	+= ../data/stage5song.inc,filter:0,offset:11776
		Run time: 32 seconds
		Input data size: 2242
		Output data size: 1765(+256 loop +27 pad)
		Average bytes per frame: 1.052604
		Average bytes per second: 63.156250

	+= ../data/titlesong.inc,filter:0,offset:13824
		Run time: 21 seconds
		Input data size: 1957
		Output data size: 1574(+0 loop +474 pad)
		Average bytes per frame: 1.249206
		Average bytes per second: 74.952381

	+= ../data/welldonesong.inc,filter:0,offset:15872
		Run time: 25 seconds
		Input data size: 1827
		Output data size: 1309(+256 loop +483 pad)
		Average bytes per frame: 1.018217
		Average bytes per second: 61.093038

	== 7 entries total, 17920 bytes file size
It writes songs contiguously after the previous one(which itself might be padded out to a 512 byte boundary). It stores 32 bit offsets for the start of each song, starting at the specified offset and moving updwards 4 bytes per entry. Allowing the start of the directory and the song data to be specified should allow this to cooperate with other techniques, such as having a directory and data for "streaming ram tile graphics" or similar. It also appears to operate correctly for C array output, which will be useful for direct reading of compressed music with no buffer(to be implemented).

That is all for now, SPIRamMusicDemo will be updated to call this from the makefile and then finally(once I get the weird bugs unrelated to mconvert sorted) there is a fully working demo for all aspects of the process.

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by nicksen782 » Sun Oct 08, 2017 2:39 am

Looks good, looks clean! Before I was using a bash script that would loop through a list for the midiconvert commands and then through a list for the mconvert commands.

So, now mconvert is driven by a config file and doesn't require midiconvert to be used first? I like!

Mconvert now outputs directly to a binary file instead of C arrays? I was using the C arrays in my C2BIN program. But, since the mconvert is already binary I could just include it in the game's files. Once game reads in the SD card files it really won't matter so long has I have the correct SPIRAM starting offset, right?

I had a short song that would crash mconvert (not midiconvert.) How does mconvert handle errors now?

Great job man!

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Sun Oct 08, 2017 5:01 am

nicksen782 wrote: So, now mconvert is driven by a config file and doesn't require midiconvert to be used first?
Yes! I feel like this is probably the most important development in streaming music, since now it is actually realistic to use instead of diving through cryptic notes and source code to even figure out how to get the data. mconvert does require midiconvert to create the data from the original MIDI file just like normal, or you can of course take preconverted songs.
nicksen782 wrote:Mconvert now outputs directly to a binary file instead of C arrays?
It can do either now. I said I wasn't going to for a while, but it wasn't too bad so I added C array output back. I have tested it and feel the output should be correct. You can take a quick glimpse at the output style, if you open up the data/sdImageConfig.cfg in the SPIRamMusicDemo, and change the first value to 0 to force flash output. You should probably change all those "NULL" to some sane array name too. If you change something in the configuration file then rebuild SPIRamMusicDemo, it should automatically rebuild the music file. Testing so far shows that C array output works correctly, no loop buffers or padding as those are useless for the flash only version. Still have to develop the actual bufferless flash version of course, but right now anyone can drive the buffer directly from PROGMEM if they can spare about 10 bytes of ram for now; no need for SD or SPI Ram.
nicksen782 wrote:Once game reads in the SD card files it really won't matter so long has I have the correct SPIRAM starting offset, right?
If I understand it correctly, you are correct. Essentially in the examples, there is a 4 byte value which starts at offset 0 in the file. That value can be read by SpiRamReadU32, and you can use that value then directly to cue up music like "SpiRamSeqReadStart(songBase>>16,songBase&0xFFFF);". The value at 0..3(in the examples, you can start at places other than 0) is the offset for the first byte of the first song. The 4 bytes at 4..7 is the offset to the first byte of the second song, etc. The conversion program assumes that you are loading the entire thing into the bottom of SPI Ram, but if you were doing something else, you could just add a fixed value to the directory values you read to get to the data even if you move it around. So at the very least they are a relative offset no matter what you do.
nicksen782 wrote:I had a short song that would crash mconvert (not midiconvert.) How does mconvert handle errors now?
I made some improvements on preventing bogus strings or other things from crashing it, but I am sure it is not crash proof by any means. I would be very interested to know if the new version also crashes with the song, and if so, I would appreciate a copy of that file so I can figure out what is going on there. There are some error checks during the conversion process, but there are probably a couple assumptions made that could be improved. The primary goal is to never crash for any possible output midiconv can create.
nicksen782 wrote:Great job man!
Thanks man! For the continued interest and feedback as well :)

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by nicksen782 » Sun Oct 08, 2017 6:55 pm

Makefile for SPIRamMusicDemo runs mconvert like this:

Code: Select all

SD_MUSIC.DAT: ../data/sdImageConfig.cfg
	$(UZEBIN_DIR)mconvert ../data/sdImageConfig.cfg
However, you need to preceed with a "./". UZEBIN_DIR is not defined either so it ends up blank. Mconvert must be compiled first and if on Linux would need a chmod +x mconvert also.

Mconvert ran into an error with ../data/stage56song.inc ... But it had the "#" for being commented out. Seemed the comment command is being ignored. I put a space between the "#" and the rest of the line and the commented line was ignored.

After that everything started compiling as expected but I ran into that "undefined reference to `songStalls`" error again. I see that it is declared extern with extern volatile u16 songStalls;. Unfortunately I see it referred nowhere else in the streaming-music branch files. Sorry.

Progress is still good. I want to play with it! ;)

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by nicksen782 » Mon Oct 09, 2017 1:38 am

Okay, the new code appears good (aside from that songStalls thing.)

The included music plays well.

So, I tried with my own. It turns out that mconvert does NOT read raw .mid files. It still requires midiconvert to run first. So I did that. Then I edited the sdImageConfig.cfg file to make the first song my test song. It plays super fast!

The songs included with the demo however play correctly.

I've included the folder zipped up. You'll see the original .mid file and the output from midiconvert.

This is how I used midiconvert:

Code: Select all

./midiconv streamingMusic/org/m_01_Title.mid             streamingMusic/m_01_Title.inc             -v m_01_Title            
Am I doing something wrong with midiconvert? Is the base .mid messed up?
Attachments
SPIRamMusicDemo_tooFast.zip
(25.22 KiB) Downloaded 80 times

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Mon Oct 09, 2017 3:40 am

I am going to guess that the fast playback is already there before mconvert touches it. The reason I assume that, is that almost always I need to specify a speed correction factor to get my songs to be the right speed. The flag is "-f", and if you don't specify it, it defaults to 30.

So if it is playing say 50% too fast, I would recommend doing something like this:

Code: Select all

./midiconv streamingMusic/org/m_01_Title.mid streamingMusic/m_01_Title.inc -f45 -v m_01_Title 
If it is still too fast, try "-f50". If 45 becomes too slow, then try something like 40. The larger -f is, the slower the song should play, and vice versa.

Eh, converting MIDIs in itself, can sometimes be quite a process to get things dialed in right. There are no plans to implement raw .MIDI reading as it would make the code base much larger and be quite a lot of work, when midiconv already does it well.

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by nicksen782 » Mon Oct 09, 2017 5:04 pm

I've just been finding songs on the Internet. I'm starting to think that this is more to do with the midi itself.

I'm also wondering about the patches used.

I've attached a working copy that has 2 songs on it. The second one sounds perfect. The first is just crazy fast. I used -f 3 on it. -f2 was too slow and -f 4 was too fast.

How could the midi be fixed? What about the 01 title midi from my last attachment?
Attachments
anothertest.zip
(16.58 KiB) Downloaded 89 times

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Mon Oct 09, 2017 5:54 pm

I will check this out later tonight. Some times the ticks per second is so jacked up that it does not divide nicely, and so midiconv can't offer fine tuned enough between 3 and 4 for instance. That is my understanding, though things have changed with midiconv since most of my music development. Possibly changing the ticks per second will allow finer control with -f. I will look, you can sort of tell what is going on by seeing the event list. Some songs have really large tick numbers there which might affect it.

I did at least once some how manage to fix a song by creating a new one, and copying the data from the MIDI into the fresh one. I had to screw around with tempo and I remember that whole process sucked and took several tries. I have given up on some MIDIs and found an alternate source because it was easier. If it is from NES,SNES,Genesis and others, you can extract it yourself and usually those work cleanly, but especially SNES and Genesis, will require some choices on what channels to delete, what channels to mix, etc. It can be a process. If you absolutely need a specific MIDI, let me know.

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by nicksen782 » Mon Oct 09, 2017 7:26 pm

I was using Anvil Studio to change a few things in that Metroid midi. I figured if I was going to be learning a program and hearing the same song repeatedly it might as well be that swingin' Brinstar theme!

I lowered the play speed. Had way more affect on the midi playing on my computer than it did midiconvert. Also, ugly sounds! This is the patches part right? The midi has different instruments and the patches are the instruments? Help?

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest