SD Image Creation Tool

From Uzebox Wiki
Jump to navigation Jump to search

Introduction

These tools are intended to save you time and self hair-pulling-out. One of the general problems that is sometimes more difficult than the code itself, is to get data into an SD image that is the correct format to be loaded and used by your program. One solution I have used for years, is to manually build the data using a combination of copy/paste and in place editing inside a hex editor...that is every bit as fun as you might imagine ;) The real issue, that is a huge waste of time that is not focused on the fun parts of development. It is also error prone, and potentially a huge pain if formats need to change later or resources need to change size or move to other locations. So far my games done manually, do indeed have spurious information and wasted space as remnants of things moving and formats changing as the game evolves. That leaves a large temptation to leave empty voids between data for future changes, and for bigger things, this space waste adds up even for the size of an SD card.


With the advent of Streaming Music, it became very important to have an automated way to build the data, as often times music will change quite a bit from it's initial conception, just as graphics will. Tack that on with the fact that many will likely want to use SPI Ram to offload multiple types of resources(graphics, music, etc.), there is a clear need for automatic and fast creation of these images. Now, some power users might note that all this could likely be done with good usage of built in things like "cat" on Linux with perhaps some scripting. That is also valid and perfectly good, but not necessarily cross platform or common knowledge. Also, simply having a small tool with easy to understand source code and built for more specific tasks, can have an advantage. And so here it is, since I found it's creation necessary, to save my sanity!!


What It Is And Is Not

This is designed to be a very simple set of tools, that do what they need to do, and nothing else. They are generally agnostic, and do not know at all, what this data is nor do any modification(except for streaming music compressed format). There is already a good and powerful selection of tools available for graphical, music, and SFX generation and there is no sense in duplicating that functionality. These tools are designed to harness the direct output of these well established programs, and performing only mild formatting, turn it into usable data in an SD image without you ever having to open up a hex editor to manually intervene. Then you can focus on programming which is productive and fun.


hex2bin

This tool does what it's name implies. If you have a file with a C array, you can turn this into a binary representation and directly write it to a file, at any position you choose. You can optionally also have it write the offset that the data was written to, at another location. This is designed to make creating directories which contain pointers to find the data, easy. In this way, your program can find different objects/entries, without having to hard code addresses/offsets.


This is rather useful for a multitude of things, as most of the existing utilities will output a C array originally intended to directly include in the program. Instead, all these tools can be used exactly as before, to build an SD image with the data. This allows all existing tools to be useful for SD games without any extra tools or arcane knowledge.


As an example let's imagine that we have used gconvert to generate tile and map data from an image, and we would like to store that ready to use(perhaps in the way described in Flash Free Screens From SD) on the SD card. In order to keep the source code simple, and the command line usage simple as well, it can only do 1 array at a time, but you can choose the array. In gconvert imagine you have tile data followed by map data. Let's first put the map data somewhere, and put a pointer to it in the directory so our program can find it:

./hex2bin gfxin.inc SD_IMAGE.DAT -o512 -d0 -z3

This will put a binary representation of the first C array in the file(the map data), starting at offset 512 in the file "SD_IMAGE.DAT". If the file exists, it will not destroy it, and will overwrite anything starting at offset 512 up to the size of the data, expanding the file if necessary. If it does not exist, it will create it. It will also write a 24 bit(3 byte) value at offsets 0..3(MSB first) that is the offset of that data. This directory entry can be written anywhere(specified by -d0 here), and be any size(specified by -z3 here), though 24 is likely sufficient for most cases. So in this case starting at offset 0 in the file, we would find

0x00,0x02,0x00,//24 bit MSB first hexadecimal representation of decimal 512, the offset we wrote the map data to

Now if we also wanted to put the second array in the file "gfxin.inc" some where else, we would call it again like:

./hex2bin gfxin.inc SD_IMAGE.DAT -o2048 -d3 -z3 -k0

Again we are using a 3 byte/24 bit directory entry specified by "-z3"(could be "-z2", "-z4", for 16 bit or 32 bit entries, any format you like), and we are putting that directory entry at offset 3 in the file(because the first entry used bytes 0..2). We are putting the tile data after the map data we wrote before, at offset 2048 specified by "-o2048". If you know the final size of vram you will use, you could pack tighter, we are just leaving some space in between for now. All of this of course is read from "gfxin.inc" and is written to "SD_IMAGE.DAT" like before. We are also specifying "-k1" because by default the program will read the first C array it finds(which was the map data), perform the transform and output, then quit. Since we already did that in the previous step, we tell the program to skip the first array it finds, which will cause it to read the second; in this case that is the tile data. Doing these 2 steps we have total control of putting any graphical data wherever we want, as well as simultaneously creating a directory so our program can find that data. This example has the techniques found in Flash Free Screens From SD and Flash Free Screens From SPI Ram in mind, though they could be used for any other technique you might require.


Another example of using this technique for music, as well as using the "mconvert" music compression utility, can be found at Streaming Music.


hexcrunch

This is another simple tool aimed at providing simple compression to data in an automatic way, which has output suitable to be used with hex2bin in creating SD images. It can also be used for data intended to directly include in a game, instead of on an SD image. There are no efforts to support advanced compression which requires complex source code and ever more specific formats. Instead it supports just 2 basic types of compression, and a couple submodes.

The first type of compression is the good old fashioned RLE, which is a fast and simple compression scheme which works well for some types of data. For a quick tutorial covering this concept and others, check out Compression Tutorial. There are a large amount of possible variations to the scheme. This program supports:

mode  0: RLE - 1 byte data, 1 byte length
mode  1: RLE - 2 byte data, 1 byte length
mode  2: RLE - 1 byte data, 2 byte length
mode  3: RLE - 2 byte data, 2 byte length
mode  4: RLE - 7 bit data, 1 bit length
mode  5: RLE - 6 bit data, 2 bit length
mode  6: RLE - 5 bit data, 3 bit length
mode  7: RLE - 4 bit data, 4 bit length
mode  8: RLE - 3 bit data, 5 bit length
mode  9: RLE - 2 bit data, 6 bit length
mode 10: RLE - 1 bit data, 7 bit length

Note that when encoding a length, it is implied that there is a minimum length of 1. So in mode 0, the 2 bytes 0x55,0x00 indicate a single value of 0x55. For the values 0x77,0x03 it would indicate a string of 4 bytes each having the value of 0x77. The same applies to length in the bit modes.


It also support bit packing, which is taking the values of multiple bytes and putting them in a single byte. This can only work, if the values do not require all the bits in a byte. The modes are:

mode 11: bit pack - 1 byte to 1 bit
mode 12: bit pack - 1 byte to 2 bits
mode 13: bit pack - 1 byte to 3 bits
mode 14: bit pack - 1 byte to 4 bits

To help understand, as an example the string of 8 bytes 0x00,0x00,0x00,0x01,0x01,0x01,0x01,0x01 would compress to single byte with the value 0b00011111 under mode 11. Under mode 12, the same string would compress to the 2 bytes 0b00000001, 0b01010101. Under mode 14, the string of 4 bytes 0b00000001, 0b00000011, 0b00000111, 0b00001111, would compress to the 2 bytes 0b00010011, 0b01111111. Bits are stored from left(most significant) to right(least significant).