The UZE file format

Topics on software tools like TileStudio, comments on documentation and tutorials (or the lack of) should go here.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

The UZE file format

Post by Jubatian »

I added a new Wiki page to describe the UZE file format:

http://uzebox.org/wiki/UZE_File_format

I think such material didn't exist yet, at least no matter how hard I tried, I couldn't find it. I assembled the description using the 0.4.5 bootloader's code and Packrom's code.

Now that it exists, it could be nice to add some extras for supporting extra data. I think about two important elements which could be nice to have:
  • Sectors reserved for Save states for games which wish to use the SD card to store saves without needing anything complex
  • Sectors reserved for additional game data, would be particularly useful for SPI RAM games
Both of these are intended for situations where there is no much reason in creating separate files (so not for games which may use user-made content, for example stuff like extra levels). Of course any extra data is to be placed beyond the end of the game, so the UZE files can be handled normally with any existing program.

For save states, I would add that the default state should be having this region zero. This way, just like the EEPROM, the saves could be wiped in games without the knowledge of the game's inner workings, which can be a useful feature. Games can easily support this for example by having a nonzero marker indicating that the save is present.

Without this, I could imagine situations where you end up with a game carrying some unwanted save state which you have no way to clear, for example if even the "empty" game has some complex binary structure in the save region.

Having a save state in the UZE file is not a risk in my opinion. A game writing the SD must know where it wants to write, so if it is screwed, it doesn't matter much whether it is screwed within the game file or would barf out something somewhere else on the card, it could be just as bad. You have to be cautious if you are writing.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: The UZE file format

Post by Artcfox »

Thanks for writing this up!

For a 32x32 animated icon, I think we could have special magic bytes written to the 16x16 icon field and a pointer to the 32x32 animation, and length.

My preference would be for the. uze files to remain read-only, just because I copy them around all the time, and having game state come with them might be bad. I'm thinking the online emulator, or testing games before shipping a rom pack. Keeping save state in a separate file would take no more code (you still have to open a file by name, and calculate write point, and write a sector), and it avoids so much hassle (Now when sending aomeone all the ROMs I have on my system I have to remeber to zero all the save states?)
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: The UZE file format

Post by D3thAdd3r »

That was needed, good job writing that up.
Artcfox wrote: Sat Apr 14, 2018 12:58 pm For a 32x32 animated icon, I think we could have special magic bytes written to the 16x16 icon field and a pointer to the 32x32 animation, and length.
I would very amused if we could actually utilize this, as I generated quite a few icons for Emuze for different games which could double over. That I recall, Alec said the original bootloader was packed to almost the last byte which Sandor also stated about the new bootloader. I am definitely in no place to make suggestions about implementation for either of those, but if it were somehow possible to do it would be a very cool feature even at 16x16 I feel.
Artcfox wrote: Sat Apr 14, 2018 12:58 pm My preference would be for the. uze files to remain read-only, just because I copy them around all the time, and having game state come with them might be bad.
This might make the most sense and probably most SD games wont write anyway. I am only aware of Toorum's Quest editor which writes on it's own data file, and that could just as well stay a separate file(having completed the editor and releasing 1 very hard episode, I became bored to develop more of them). In a certain way I guess the author of each particular game knows best, but in general I agree. It does seem a bit "impure", but excessive files on the SD card is also annoying. In TQ for instance, there is not really game state since you start from the start each time, and the only thing you are hauling around is new episodes.

I mentioned this a while ago, but I am really convinced something like this can work well. Toorum's Quest does use that, and now that I remember that, I was going to make AE do it too so I will for all things I do further anyway. I would be interested to discuss anything involving that so that it can be modified to fit everyone's purposes. The main motivation is to eliminate excess files on the SD card. There were possible arguments against it, but mainly it shares the same "risks" as writing to the SD in general, that a runaway bug can screw everything. Ultimately I never saw that as a compelling reason not to write, versus the interesting directions writing opens up, since no real life critical information should ever be put on Uzebox. Something like Pacman long ago had a bug, which would break the whole EEPROM. A couple users got broken saves on accident, not a big deal, it was fixed. Probably a lot of games directly copied the fixed code for interfacing with EEPROM ever since. Anyway...I will stop my ideological push here since I think most agree writing is an interesting new thing.
Jubatian wrote: Sat Apr 14, 2018 8:49 am Now that it exists, it could be nice to add some extras for supporting extra data. I think about two important elements which could be nice to have:

Sectors reserved for Save states for games which wish to use the SD card to store saves without needing anything complex
Sectors reserved for additional game data, would be particularly useful for SPI RAM games
If totally pragmatic about it, save sectors could have some benefits versus the concept of not keeping state in the game which I also understand. I guess I am not sure which "side" I am on right now, since I can see pros and cons for either.

Sectors for additional game data--what exactly do you have in mind for this?
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: The UZE file format

Post by Artcfox »

D3thAdd3r wrote: Sat Apr 14, 2018 6:20 pm I mentioned this a while ago, but I am really convinced something like this can work well. Toorum's Quest does use that, and now that I remember that, I was going to make AE do it too so I will for all things I do further anyway. I would be interested to discuss anything involving that so that it can be modified to fit everyone's purposes. The main motivation is to eliminate excess files on the SD card.
Oh wow, this is the first that I've seen of that! I think this is probably the best idea, have a file that has reserved blocks similar to the EEPROM, and if people are using the bootloader's read/write support with CRC then it shouldn't be that risky. To eliminate almost all risk, I could even envision a SDSaveSector API where you just tell it the reserved sector number and give it a data pointer and a length, and let the tiny optional library handle everything. That way everyone who wants to use it can just use the same standard code (hopefully robust without bugs) for accessing it.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: The UZE file format

Post by nicksen782 »

A reserved sector sounds great considering the gain of having no file system overhead. However...

You would need to read/write a specific sector from the SD but you typically don't have that kind of choice under normal circumstances when copying files. How can you make sure that something is not already written to that sector already? And once written it is there in that sector but it isn't part of the file system and will still be reflected as free space. Any further writes to the SD card run the risk of overwriting that sector.

This sounds a bit more complex than it is useful. Am I not understanding something?
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: The UZE file format

Post by CunningFellow »

@nicksen - I think they mean a sector reserved within the address space of the file - not the absolute address space of the CD.
D3thAdd3r wrote: Sat Apr 14, 2018 6:20 pm That was needed, good job writing that up.
Artcfox wrote: Sat Apr 14, 2018 12:58 pm For a 32x32 animated icon, I think we could have special magic bytes written to the 16x16 icon field and a pointer to the 32x32 animation, and length.
I would very amused if we could actually utilize this, as I generated quite a few icons for Emuze for different games which could double over. That I recall, Alec said the original bootloader was packed to almost the last byte which Sandor also stated about the new bootloader. I am definitely in no place to make suggestions about implementation for either of those, but if it were somehow possible to do it would be a very cool feature even at 16x16 I feel.
Jubatian is a coding machine and does amazing work and more impressively does it so fast. So this is not a comment about any ones skills, but I have had a look at the current bootloader and I think there is enough space could be saved to implement the animated icon idea.

It would require a change in the video mode to have monochrome text plus full colour RAM tiles.

Animating the icon for the game then would simply require having a 32x32 pixel of RAMtiles drawn to the screen. Then when the game is highlighted in the menu a series of SD sector reads into the RAMTile memory would be cycled through.
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: The UZE file format

Post by D3thAdd3r »

Artcfox wrote: Sat Apr 14, 2018 7:53 pm To eliminate almost all risk, I could even envision a SDSaveSector API where you just tell it the reserved sector number and give it a data pointer and a length, and let the tiny optional library handle everything.
That seems ideal.
nicksen782 wrote: Sat Apr 14, 2018 9:18 pm You would need to read/write a specific sector from the SD but you typically don't have that kind of choice under normal circumstances when copying files. How can you make sure that something is not already written to that sector already? And once written it is there in that sector but it isn't part of the file system and will still be reflected as free space. Any further writes to the SD card run the risk of overwriting that sector.
I am not sure if your concern is for the shared save file or for the .uze writing, but neither one should be any huge problem. I think everyone agrees changing file sizes is not something you do in a game, though perhaps a separate rom/utility with lots of free ram could pull this off. For something like the shared save file "_HISCORE.DAT", there is no issue since you first use the file system to find the first sector of the file you are going to write to. Then there is a fixed file size/number of sectors that makes up the file, so offset from the start of the file according to the format. When the assumption is made that there is no fragmentation, you can know exactly what sectors are part of the file on the card itself without having to walk through the chain. Use any of those sectors as arbitrary reserved sectors for whatever the purpose.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: The UZE file format

Post by Artcfox »

D3thAdd3r wrote: Sun Apr 15, 2018 1:30 pm
nicksen782 wrote: Sat Apr 14, 2018 9:18 pm You would need to read/write a specific sector from the SD but you typically don't have that kind of choice under normal circumstances when copying files. How can you make sure that something is not already written to that sector already? And once written it is there in that sector but it isn't part of the file system and will still be reflected as free space. Any further writes to the SD card run the risk of overwriting that sector.
I am not sure if your concern is for the shared save file or for the .uze writing, but neither one should be any huge problem. I think everyone agrees changing file sizes is not something you do in a game, though perhaps a separate rom/utility with lots of free ram could pull this off. For something like the shared save file "_HISCORE.DAT", there is no issue since you first use the file system to find the first sector of the file you are going to write to. Then there is a fixed file size/number of sectors that makes up the file, so offset from the start of the file according to the format. When the assumption is made that there is no fragmentation, you can know exactly what sectors are part of the file on the card itself without having to walk through the chain. Use any of those sectors as arbitrary reserved sectors for whatever the purpose.
Fragmentation shouldn't be a problem, as long as the bootloader's FS library is used, though it is my understanding that the fallback library does not support FAT32, so properly supporting fragmentation might be a concern if the _HISCORE.DAT file is on a FAT32 volume that is fragmented, and the new bootloader isn't installed (but if the new bootloader isn't installed then the user wouldn't be storing roms on a FAT32 volume in the first place, so it might be a moot point.)
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: The UZE file format

Post by nicksen782 »

I understand now. That sounds like a great idea. Sectors within a file. So, you look at it as a sort of file system where each file is 512 bytes and the start byte is a reserved multiple of 512? As long as the main file can be found that sounds pretty cool.

I think writing to the .uze file is an unsafe idea. How would you know if you have a clean rom? The only real argument I can have for it has to do with slack space at the end of a file. A file can consist of many clusters but a cluster can only have one file. If there is space at the end of the cluster which would be wasted anyway then sure, go ahead and use it. But then you would need to be aware of the cluster size which can vary. And, if you write beyond the slack space in the cluster then what about fragmentation? The very next cluster may not be available.

Or, for SD Cards perhaps it is more important to consider sectors. Same argument applies.

I think that SD writes should be reserved for dedicated files, shared or included files with the game. Keep it less complex. If I copy a game to an SD card and it only works sometimes then I would be upset. Ultimately, things need to work. Sometimes you just want to play a game and not monkey with it for an hour to get it to work.

Icons are cool if they can be put there. If that could work that would be really cool.
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: The UZE file format

Post by D3thAdd3r »

A big part of this is probably the "SPM Hijacking", where it could potentially be easy for any member without an ISP to get to the same state with the new bootloader. At least then it seems not that bad of an assumption to make that the new bootloader is there. Partially I am for the idea of a game relying on the bootloader since there are code space benefits. Then if there are issues there is really no problem with the game code, it is a problem with the user, not having upgraded his bootloader :P If it requires an ISP as it does now, it is not so easy take that attitude.
nicksen782 wrote:But then you would need to be aware of the cluster size which can vary. And, if you write beyond the slack space in the cluster then what about fragmentation? The very next cluster may not be available.
Eh one of many reasons I am adamantly against expecting a game to cope with it. At least IMHO, it is not worth spending game resources or limiting what you do with SD for fear of upsetting the cruel mistress that is fragmentation.
Post Reply