mconvert - MIDI Compressed Format Tool

Topics on software tools like TileStudio, comments on documentation and tutorials (or the lack of) should go here.
User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Mon Jul 03, 2017 3:49 am

I am creating a thread for this for any longer term development on this tool, which is currently only in the streaming-music branch on github. I expect this to be rather minimal thread, but here it is in case there are questions, bugs, or feature requests specific to this and not the streaming music player in the kernel. The majority of the initial development was documented here.

The streaming music at this point is in an initial state of working and that, as well as this tool specifically, should be ready for use in development. Because the streaming music was intended to be agnostic to where the data comes from, nothing specific to SD(the expected most common source) is necessarily implied in the kernel or mconvert. In SDMusicDemo I just determined the end of the song when the kernel found and set "loopEnd". Since it was stored statically on the SD that works, but trying to make a SPIRamMusicDemo I realized there is not a good way to know how much song data there is, without processing through the entirety of the song and being aware of all the format details the song player is. This seems quite a limitation considering the intended goal is being agnostic to where the data comes and to be easy to use in different scenarios.

I added a new little option "-s" to prepend a 16 bit value that is the length of the output song. This is so that one can read those 2 bytes first, to know how much data follows. In general this might be most useful for SPI ram where you determine how much to load, read that much from the SD card into SPI ram, and then stream it from SPI ram. In fact, that might be the only need for it. So that's it, maybe nothing else is required from this little tool, time will tell.

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Mon Jul 03, 2017 7:07 pm

Some discussions with nicksen782 on Zelda have me thinking about what it is like to implement multiple streaming songs in a game. This is the most common case, as otherwise you would probably just throw it in flash with the stock player.

There are a couple methods, which the existing SDMusicDemo does the simplest and highest performance one. It knows exactly where the song starts(offset 0 in the file) so seeks there and starts reading. If I added a second song, so long as I knew where it started(since you can specify an offset to insert song data into a file) I could just keep that number in flash also. Honestly if a song works and is tested through the normal midiconv process, it should sound exactly the same as normal so maybe this solution is not so bad.

Another solution would be to store pointers somewhere in the file which is a directory to the start of each song. I don't think this is anything the kernel should dictate or do, as it is going to be so different depending on the needs of the program or the source of the data. Though, maybe it is easier to keep updated, I question how much it needs to be updated after a song is developed. This has the disadvantage of requiring a seek to the directory, reading 2 bytes, then seeking to the offset you found, instead of jumping to a known offset in flash.

Talking that through, I generally lean to the less fancy concept of just storing it in flash but the other way would work too. Like I say, I don't intend to have the kernel do anything with that. But if someone wanted to do it this way, perhaps mconvert can have another option to help with this? Any ideas how that should lay out? I was thinking since you can already specify an offset to put the data, there can be another flag to specify a location to drop a 16 bit representation of that offset(where you put the song data). Then you could build the directory anyway you like just having to make sure all your arguments are good. That is the best generic solution I can think of, is it even worth it?

User avatar
nicksen782
Posts: 667
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by nicksen782 » Mon Jul 03, 2017 9:43 pm

If you can seek to the beginning of a file you would be on a 512 byte boundary since files should always start on a 512 boundary. Zelda has one datafile and you seek to anywhere in it. I have it set so that it gets to the correct sector of what you want to read and then it skips bytes until it gets to the desired bytes. For me, this occurs when the screen transitions and if it brings a delay I'm not that aware of it. However, how often would this byte skipping need to take place with the SD streaming music? I don't know how the demo does it but would I be correct in assuming that it just keeps the SD file open and reads bytes? This would negate further byte skipping. I certainly would not be using it that way.

How big are the individual music files?

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Tue Jul 04, 2017 12:38 am

The size of each one will vary quite a lot, but most likely in the range of 20-40% smaller than whatever the original was.

If someone knows better on SD stuff, correct me, because I am no expert. Basically at the SD level there is no such thing as files, just sectors. Some sectors store data about the file system indicating where to find such things, but that information on a sector is used as a higher level construct which would make something abstract you'd think of as a file. So there is no such thing really, as an open file here, just an in process multi-byte read and an original location you looked into the file system sector to find the start of. Say if you are reading song bytes to fill the buffer, then next frame you need to grab data to fill vram and ram tiles. I think you already know this but you must do a sdCardStopTransmission() to stop that song reading. Then you need to seek to the location of the vram and ram tiles data you want, and read that. When you are done with that, you need to sdCardStopTransmission() again, and seek back to the song data. The SD card cannot resume some place you used to be, because it is already operating at it's lowest level efficiency possible with direct sector access. Switching back and forth is very costly, but hopefully it happens within a frame or 2 which you can buffer for.

I would recommend instead of seeking a sector then manually skipping bytes, to use sdCardCueByteAddress() which will do the same thing internally. It will get to the sector initiating a multi-byte transfer, and if the address is inset it will eat bytes until the next byte is the one you want. It is all done in assembly already, so it seems more difficult and slower to do it in 2 steps with C.

User avatar
Artcfox
Posts: 944
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by Artcfox » Tue Jul 04, 2017 3:02 am

Just to add something, that I'm not sure is implemented with the latest SD card routines (in order to save space), but would be an easy add. As far as I know all of the reads are done as CMD18 streaming reads, which requires a CMD12 (stop transmission) to end them, which takes a while for it to stop. If you know that you're going to be jumping around sectors every few frames then I think it would be faster to just use a CMD17 (single sector read) for this, because when it's done, it's done, and you don't have to issue a CMD12 and then wait for the transmission to stop.

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Tue Jul 04, 2017 5:09 am

That seems a good idea if it works out for the level/screen reading stuff even if there is a bit of over-read to ram tiles or something. I am wondering then, does SD always need byte address?

Code: Select all

sd_card_cue_sector_address:
    mov     r25, r24                        ; Regular SD needs byte adress so shift sector value by 9 bits (*512)
    mov     r24, r23                        ; by first shifting it one byte (8 bits)
    mov     r23, r22
    clr     r22
    lsl     r23                             ; and then follow up with 1 more shift for a total of 9 bits
    rol     r24
    rol     r25

sdCardCueByteAddress:
sd_card_cue_byte_address:
    movw    r20, r22                        ; shift the address from R22:23:24:25 where it came in
    movw    r22, r24                        ; into R20:21:22:23 where sd_card_send_command expects it

    ldi     r24, SD_CMD_READ_MULTIPLE_BLOCK
    rcall   sd_card_send_command

    rcall   sd_card_get_data_token          ; wait for data token
    cpi     r24, SD_DATA_TOKEN
    breq    sd_card_cue_end                 ; if data token received cue_sector succeded

                                            ; other wise there was an error and we need to

                                                       ; return fail and
    rjmp    sd_card_clock_and_release_with_error       ; release the SD card bus
    ; phantom ret

sd_card_cue_end:
    sts     sd_512_byte_count+0, r1         ; Reset the 512_byte counter
    sts     sd_512_byte_count+1, r1
    clr     r24                             ; return success
    ret
It seems like it works the opposite way I said before actually. In either case it seems best to use sdCardCueByteAddress() for restoring the old position when swapping back and forth, and best to finish the level loading code in 1 shot where possible.

User avatar
Artcfox
Posts: 944
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by Artcfox » Tue Jul 04, 2017 5:16 am

D3thAdd3r wrote:I am wondering then, does SD always need byte address?
During initialization you must query the card, and that's where you check to see if the CCS is set or not. If it's not set, it's an SD card, and uses byte addressing. If it is set, then it's an SDHC/XC card and you must use block addressing.

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Tue Jul 04, 2017 5:26 pm

Do you know if it already supports SDHC then and also is SDHC going to be lower performance for our purposes(yes I think)? I know the bootloader cannot currently flash of an SDHC...even though I have a 2GB card labelled SDHC(weird..) that works, my 4GB non-SDHC(took a while to find such a thing) sort of works sort of doesn't, and cannot run Alter Ego or T2k properly. But in AE case that could just be a Petit FatFS issue which I will probably go back and do simpleSD and streaming music.

User avatar
Artcfox
Posts: 944
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: mconvert - MIDI Compressed Format Tool

Post by Artcfox » Tue Jul 04, 2017 10:53 pm

I'm not sure of the SDHC initialization support made it into the current bootloader. CunningFellow would know for sure. When I wrote a SD/SDHC/SDXC library, I noticed less delay between sectors on the SDHC cards vs SD cards, but I didn't test the performance if the SDXC cards.

User avatar
D3thAdd3r
Posts: 2422
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: mconvert - MIDI Compressed Format Tool

Post by D3thAdd3r » Wed Jul 05, 2017 5:56 am

Alright I implemented "-d" which can be used as some sort of directory if anyone wants to do things that way. Essentially if you are in binary mode, and you specify, or don't, an output offset, you can do something like "-d 0" or "-d 4" or "-d 8". It will output a 32bit value representing the offset the song was placed in the file, to byte offset 0, 4, 8, etc. What this can be used for, is to have your first song output wherever and do "-d 0", and the first 4 bytes of the file would be whatever offset you put that song. Maybe your second song some other place, and "-d 4". Past this I can't think of too much functionality in that direction that would really make sense, so this might be the last feature added.

I will put it up on github a little later, unless anyone needs it right away let me know.

Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests