New cross-platform C++ version of midiconv

Topics on software tools like TileStudio, comments on documentation and tutorials (or the lack of) should go here.
Post Reply
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

New cross-platform C++ version of midiconv

Post by Artcfox »

I got sick of having a Java dependency, so I ported the Java tool named MidiConvert contained within the uzebox/tools/JavaTools/dist/uzetools.jar package to C++.

Its purpose is to convert a MIDI song in format 0 or 1 to a Uzebox MIDI stream outputted as a nicely formatted C include file.

The code lives in the uzebox/tools/midiconv directory, and it should run on Linux, OS X, and Windows (using mingw).

It is built using Midifile, a really awesome open-source C++ library for parsing Standard MIDI Files (c) 1999-2015, Craig Stuart Sapp.

To speed up the build time, all of the original examples that were in the src-programs directory have been moved to src-programs-extras. Any of those files can be copied back into the src-programs directory and then built using the instructions below.

BUILD/INSTALL INSTRUCTIONS:

Code: Select all

make
or:

Code: Select all

make library
make midiconv
This will build and install the midiconv binary as uzebox/bin/midiconv at which point you can run it using basically the same options as the Java version, but pay close attention to the command line arguments because some may be slightly different (and long options are provided).

To get a list of all supported options, just run:

Code: Select all

./bin/midiconv --help

Code: Select all

Usage: ./bin/midiconv [OPTION]... INFILE OUTFILE                  
                                                                     
Options:                                                             
   -h, --help     Display this help text and exit                    
   -f, --factor   Speed correction factor (double). Defaults to 30.  
   -v, --varname  Variable name used in the include file. Defaults   
                  to 'midisong'                                      
   -s, --start    Force a loop start (specified in ticks). Any       
                  existing loop start in the input will be discarded.
   -e, --end      Force a loop end (specified in ticks). Any existing
                  loop end in the input will be discarded.           
   -d, --debug    Print debugging information                        
   -1, --no1      Include note off events for channel 1              
   -2, --no2      Include note off events for channel 2              
   -3, --no3      Include note off events for channel 3              
   -4, --no4      Include note off events for channel 4              
   -5, --no5      Include note off events for channel 5              
TIPS AND TRICKS:

If you used Rosegarden to create a MIDI file by arranging multiple segments, unless those segments are joined by clicking the name of each track (to select all segments on that track) and pressing Ctrl-J to join them, the resulting Uzebox C include file may contain multiple redundant control messages, which wastes space. After you export your MIDI (using File > Export > Export MIDI File...) you can undo these joins if you wish to keep your original segmented source file (for instance to preserve any of your linked segments).
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: New cross-platform C++ version of midiconv

Post by Artcfox »

Now that I've verified this builds and runs on Linux, OS X, and Windows using MinGW, I've added this to the main Uzebox Makefile.

It does require a compiler that supports C++11, so if you're running an ancient version of MinGW, it might require an upgrade (or you can comment out the line that adds it to the list of tools to build).

At the same time, I added a verified, cross-platform version of bin2hex (for converting raw PCM samples into C include files) to the main Uzebox Makefile.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: New cross-platform C++ version of midiconv

Post by Jubatian »

Nice, I didn't use this yet, but it's always nice to not have to lug around a hefty Java virtual machine just to do some little thing.
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: New cross-platform C++ version of midiconv

Post by D3thAdd3r »

Great tool. There is a feature I always wanted that seems easy to implement. Whenever you make an midi using nfs2midi, it generates tons of dummy expression controller events which take up a lot of space. My process is to manually go through the event list and delete these one by one, and this is usually several hundred maybe a thousand depending on the song;obviously very tedious! I never found any MIDI program that just lets you delete all events of 1 kind, so I am thinking about this section of the code in particular:

Code: Select all

        if (mev->isController()) {
          if ((*mev)[1] == CONTROLLER_VOL ||
              (*mev)[1] == CONTROLLER_EXPRESSION ||
              (*mev)[1] == CONTROLLER_TREMOLO ||
              (*mev)[1] == CONTROLLER_TREMOLO_RATE) {
            setEventTick(mev, tempo);
            newmidi[track].append(*mev);
          }
What do you think the best way to implement a filter is? I was thinking just passing a '-s0' representing binary flags for argv for none of those events, '-s3' for tremolo and tremolo rate..'-s8' for volume only, or should there be individual flags for each?

Past that, what are your thoughts on how to approach a different output format for the compressed format described here?
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: New cross-platform C++ version of midiconv

Post by D3thAdd3r »

I should mention 2 more requirements are to copy say 256 bytes after a loop start, remove the song end event (which is spurious if a loop end is specified), and put those 256 bytes after the end. The other part, pad out whatever is left so that the total output size is an even multiple of 512/sector size for easier pasting into SD resource files. And for that purpose it would need to also be something ready to be pasted into a hex editor, no array name, curly brackets, etc.

For all those reasons, is it simply better to have a post process program that reads the old current format and converts it? My thinking is another tool could exist to read binary files and insert tem at arbitrary points in another file, so that generation of SD resources (not just music) could be automated.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: New cross-platform C++ version of midiconv

Post by Artcfox »

D3thAdd3r wrote:Great tool. There is a feature I always wanted that seems easy to implement. Whenever you make an midi using nfs2midi, it generates tons of dummy expression controller events which take up a lot of space. My process is to manually go through the event list and delete these one by one, and this is usually several hundred maybe a thousand depending on the song;obviously very tedious! I never found any MIDI program that just lets you delete all events of 1 kind, so I am thinking about this section of the code in particular:

Code: Select all

        if (mev->isController()) {
          if ((*mev)[1] == CONTROLLER_VOL ||
              (*mev)[1] == CONTROLLER_EXPRESSION ||
              (*mev)[1] == CONTROLLER_TREMOLO ||
              (*mev)[1] == CONTROLLER_TREMOLO_RATE) {
            setEventTick(mev, tempo);
            newmidi[track].append(*mev);
          }
What do you think the best way to implement a filter is? I was thinking just passing a '-s0' representing binary flags for argv for none of those events, '-s3' for tremolo and tremolo rate..'-s8' for volume only, or should there be individual flags for each?

Past that, what are your thoughts on how to approach a different output format for the compressed format described here?
So are these dummy controller events duplicates of a previous controller event? I did notice that when using Rosegarden, if I composed the MIDI using multiple sections that I could drag and drop around and copy (linked) to different places, that at the beginning of each block it would generate a whole slew of unnecessary control events that were duplicates of previous control events. The way that I got around that was right before exporting to a MIDI from Rosegarden, I would select all the sections in each track by clicking on the track name, and then iht Ctrl-J to join all of its sections into one. After doing this for each track, I would export it to a MIDI, and then before saving, undo those joins to keep all the linked sections as individual unjoined sections in my Rosegarden project file. This saved me over a thousand bytes IIRC.

I had thought about implementing a filter that would detect and remove duplicate control events automatically, but I don't feel like I have a complete grasp of the MIDI format yet, so I didn't want to inadvertantly introduce a bug, especially when Rosegarden makes it so easy to do things.

I just checked for you. Open your MIDI in Rosegarden, select a track, and then press E to open the track in the Event List Editor, and on the left side you'll have a giant list of Event Filters that you can tick the box for showing. Once filtered, you can hit Ctrl-A to Select All, and Delete to delete all events of that type. There is even an advanced event editor if you want to really get down to the nitty gritty details of a particular event.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: New cross-platform C++ version of midiconv

Post by Artcfox »

D3thAdd3r wrote:I should mention 2 more requirements are to copy say 256 bytes after a loop start, remove the song end event (which is spurious if a loop end is specified), and put those 256 bytes after the end. The other part, pad out whatever is left so that the total output size is an even multiple of 512/sector size for easier pasting into SD resource files. And for that purpose it would need to also be something ready to be pasted into a hex editor, no array name, curly brackets, etc.

For all those reasons, is it simply better to have a post process program that reads the old current format and converts it? My thinking is another tool could exist to read binary files and insert tem at arbitrary points in another file, so that generation of SD resources (not just music) could be automated.
Hmm, this seems like something specific to the streaming sound engine, that may be better as a separate tool that can do the transformation, that way you can keep it in sync with any kernel changes you're making.

Personally I would just copy the midiconv.cpp file to a file with a new name, and modify it to do exactly what you need it to do. You don't even have to touch the Makefile at all. That way it keeps with the UNIX philosophy of a collection of small programs, each doing one job and doing that job well.

For the binary generation, I had to do extra work to get it to output the hex data as an array, so it should be easy to spit out the binary data how you want it. The nice thing about using the Midifile C++ MIDI file parsing library, is the massive amount of example code that comes with the library. I moved all of its example programs into a sibling directory called src-programs-extras, and if you want to build an example, just copy it back into the src-programs directory and type:

Code: Select all

make all
and anything in there will be built automtically. For your streaming specific program, just copying midiconv.cpp to a file with a new name inside the src-programs directory should have it automatically build both midiconv and your program when you type make all. If you only want to build a specific program, you can type:

Code: Select all

make midiconv
for example, and only that program will be built.

This actually seems like the perfect way to allow anyone to easily make their own custom midi conversion program without stepping on anyone else's toes, or overly complicating any one program. :)
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: New cross-platform C++ version of midiconv

Post by D3thAdd3r »

Artcfox wrote:This actually seems like the perfect way to allow anyone to easily make their own custom midi conversion program without stepping on anyone else's toes, or overly complicating any one program. :)
Oh definitely, I have no interest in muddying up an otherwise very clean program. Maybe I was just trying to get someone else to say that a separate program is a good idea, so that I wouldn't feel bad for taking the down and dirty approach...subconscious or something :roll: It is settled then, those things are a pretty terrible fit for midiconv so separate program is much nicer and no looking back.

BTW on the events I did not mean any smart filtering at all. More the type of stuff that is small and should be safe like:

Code: Select all

        if (mev->isController()) {
          if (((*mev)[1] == CONTROLLER_VOL && !(filter & FILTER_VOL)) ||
              ((*mev)[1] == CONTROLLER_EXPRESSION  && !(filter & FILTER_EXPRESSION)) ||
              ((*mev)[1] == CONTROLLER_TREMOLO  && !(filter & FILTER_TREMOLO)) ||
              ((*mev)[1] == CONTROLLER_TREMOLO_RATE  && !(filter & FILTER_TREMOLO_RATE))) {
            setEventTick(mev, tempo);
            newmidi[track].append(*mev);
          }
Even that though is kind of dirty in a sense(I am more loyal to end result than any particular ideology) if there is a way to do things like Rosegarden to filter out unwanted MIDI events. "Junk in, junk out" is a reasonable argument against such things, but it seems rather innocent if it might help someone. Somebody might even be using bloated MIDIs off the net, and not be the type who digs into the event list an all that to see why their song is massive. Either way not critical.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: New cross-platform C++ version of midiconv

Post by Artcfox »

Yeah, that type of "heavy-handed" filtering looks like a clean implementation, though I would just make them all separate named options, rather than bit-packing them, and then in the checkOptions() function you can just add:

Code: Select all

opts.define("no_vol=b", "Filter out all volume events");
opts.define("no_expression=b", "Filter out all expression events");
opts.define("no_tremolo=b", "Filter out all tremolo events");
opts.define("no_tremolo_rate=b", "Filter out all tremolo rate events");
and then to test to see if each of those options is set, you can just do:

Code: Select all

        if (mev->isController()) {
          if (((*mev)[1] == CONTROLLER_VOL && !opts.getBoolean("no_vol")) ||
              ((*mev)[1] == CONTROLLER_EXPRESSION  && !opts.getBoolean("no_expression")) ||
              ((*mev)[1] == CONTROLLER_TREMOLO  && !opts.getBoolean("no_tremolo")) ||
              ((*mev)[1] == CONTROLLER_TREMOLO_RATE  && !opts.getBoolean("no_tremolo_rate"))) {
            setEventTick(mev, tempo);
            newmidi[track].append(*mev);
          }
One line of code to define an option with an optional short form, long form, type, and default value, and one line to use that option anywhere in the code, it can't get any better than that! (Especially when compared to the C functions getopt and getopt_long, ugh)
Post Reply