The UZE file format

Topics on software tools like TileStudio, comments on documentation and tutorials (or the lack of) should go here.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: The UZE file format

Post by nicksen782 »

Writing to the header part of the .uze file doesn't sound too bad but I'm not super excited about it. I figure there is not much space anyway. In my mind, I can kinda see this as a sort of "battery backup" but I think there is a better option.

I was thinking of writing to the beginning of the data file (the .bin) of the game. Reserve space in your own file and write to it. The data file would already be there assuming you are using the SD card.

How much free space really is in the .uze file? How is it better than writing to the game's data file instead?
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

L4rry wrote: Wed Apr 18, 2018 1:41 amOK, I'm going to put this out there. A mutable binary is a dumb idea. Especially for an open platform. There is no good reason to combine your game code and assets with state.

All you will achieve is a fractured an confusing echo system whenever you share your game.
We have the limitation that the filesystem can not be changed easily (making new files, resizing files). Staying within this constraint it is an OK idea. The only real useful alternative I think would be a seperate save file with fixed size (which one's parameters could be still defined within the UZE file, to make centralized save management possible, such as to have a tolerable solution to Artcfox's earlier concern).

However that it is separate would have no much use as it has to be present (even when empty). You can not delete it to erase your save. So within this constraint (filesystem structure is fixed) I think having it within the UZE is still a better idea: it eliminates the possibility of a fault (save file missing, making the game inoperable). Think of the UZE like a cartridge for a Gameboy.

The location of the save state won't get anything safer, I mean having it in a different file isn't any safer than having it within the UZE. The breach of safety is the capability to write an arbitrary sector (and when you fail to use that in the intended manner due to a bug, no matter where you wanted to write, anything can happen to the data on the card: breaking an other game, bricking the entire filesystem, whatever). To make write safe, constrained libraries are needed which can constrain the write to the appropriate area (which demands a standardized method for saves, so such a library can be made).
D3thAdd3r wrote: Wed Apr 18, 2018 4:03 amTo be fair, I should add "WARNING THIS PROGRAM WRITES TO THE SD CARD" whenever the user selects the editor.
Could be a nice recommendation if you are using direct sector writes, especially with an experimental version of a game: so the user may know that some bug could potentially brick the filesystem on the card.
D3thAdd3r wrote: Wed Apr 18, 2018 4:03 amNow for espousing the Devil's thoughts on flash writing, we are all taking a risk running any rom off the forums since any of them could be a big virus that fries the flash in the '644.
Huh, aren't you confusing the SD and the 644's Flash here? Frying the Flash of the 644 isn't that easy, you can't do that even with the new bootloader unless you left your Uzebox running unattended overnight when the "game" decided to start killing it. The new bootloader has a 6 seconds safety delay on programming requests: you will notice it. Bricking the filesystem on the SD is a simple matter of a write going south, destroying some fundamental part of the filesystem (but even then no hardware would suffer, only your data goes up in smoke).
nicksen782 wrote: Wed Apr 18, 2018 5:20 amI was thinking of writing to the beginning of the data file (the .bin) of the game. Reserve space in your own file and write to it. The data file would already be there assuming you are using the SD card.
Larry's concern is also valid for this: You are mixing game assets with state (he says "There is no good reason to combine your game code and assets with state", in the DAT file then you would have assets and state).
nicksen782 wrote: Wed Apr 18, 2018 5:20 amHow much free space really is in the .uze file?
As much as you want. You can make the UZE larger (I mean setting the file's size, not rogue expanding from a Uzebox game), all it needs some tools to combine it with game assets (you can add stuff after the game binary in it without any existing program making a fuss about it). The game assets I think definitely should belong within the UZE where applicable (not a "must do", but should be available as an option: don't make two files where the user wouldn't have any advantage from having two files).
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: The UZE file format

Post by Artcfox »

How about a well written API function called FS_Create_File, which accepts a file name and a number of sectors, and it would create a blank (contiguous?) file of that many sectors on the SD card? Then games could create their own save file (if needed or missing) and nothing would have to ever write to .uze files, they could always be read-only.

Plus that would solve the issue of an in-game level editor being able to safely create new level files for a game.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

Artcfox wrote: Wed Apr 18, 2018 10:12 amHow about a well written API function called FS_Create_File, which accepts a file name and a number of sectors, and it would create a blank (contiguous?) file of that many sectors on the SD card? Then games could create their own save file (if needed or missing) and nothing would have to ever write to .uze files, they could always be read-only.

Plus that would solve the issue of an in-game level editor being able to safely create new level files for a game.
The problem is that it would pull in a full FAT16 / FAT32 filesystem handler, something more complex than even the new bootloader. Not that it is not possible to be done, but it would take relatively lots of ROM space which is bad. It would be nice if a game which wanted to only have something equivalent to a cartridge's capabilities (like a nonvolatile RAM on a Gameboy cart) didn't need this code.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

Based on your opinions I think currently the most accepted method would be a fixed size always present separate save file (for example GAME.UZE accompanied with GAME.SAV) for games which wouldn't want to carry a complete filesystem code just to save state. I would add its parameters to the UZE file, so the suggested fields related to the save state would still exist: this way generic tools could handle (and possibly even create) these files.

The UZE file would still be extended with a capability to hold extra game assets beyond the game binary which the game may load later (but with the above, the UZE wouldn't contain any part which could be modified assuming normal usage).
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: The UZE file format

Post by Artcfox »

Jubatian wrote: Wed Apr 18, 2018 10:29 am Based on your opinions I think currently the most accepted method would be a fixed size always present separate save file (for example GAME.UZE accompanied with GAME.SAV) for games which wouldn't want to carry a complete filesystem code just to save state. I would add its parameters to the UZE file, so the suggested fields related to the save state would still exist: this way generic tools could handle (and possibly even create) these files.

The UZE file would still be extended with a capability to hold extra game assets beyond the game binary which the game may load later (but with the above, the UZE wouldn't contain any part which could be modified assuming normal usage).
That sounds like a good compromise. And it should also leave enough room in the uze header for pointers and a size for the 32x32 animated icons (should that be implemented).

Having a standalone utility (a UZE file) that contains the FS_Create_File function sounds like a good idea, especially since it could query that field (if present) from each UZE file on the SD card and offer save file management: create, wipe, copy (say for backing up a save file to allow someone else to play a new game without losing your progress in games that only support one save slot).

If there is enough room in that utility, we might even be able to add a file defragmenter. That is something that I always thought might be fun to create.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

Okay then, so here comes a revised suggestion for the new header fields:
  • 0x193 - 0x1E2: Reserved (for further Game Description to allow making it up to 144 bytes, as I think 64 bytes isn't quite enough for this purpose if you really wanted to use this field).
  • 0x1E3: HW feature requirement flags. Bit 0: 128Kb SPI RAM required if set. (Bit 1: SNES mouse required, maybe, depreciating the SNES mouse field which one's usages are unknown to me, Whack a mole in particular doesn't have it set).
  • 0x1E4 - 0x1E5: Save state file size in 512b sector units. 0: No save state file.
  • 0x1E6 - 0x1E7: EEPROM block ID base used for game state. 0: EEPROM usage is not defined. 1: Game uses no EEPROM.
  • 0x1E8: Count of EEPROM blocks used for game state. Should be 0 (and ignored) if EEPROM block ID is 0 or 1.
  • 0x1E9 - 0x1EB: Extra data area 512b sector start offset within UZE file, Little Endian.
  • 0x1EC - 0x1EE: Extra data area length in 512b sector units, Little Endian.
The offsets take 3 bytes as the 4th byte would be always zero anyway as the FAT filesystem can not store files larger than 2Gb.

The EEPROM block ID and count of EEPROM blocks are intended to make it possible to write future-proof EEPROM management tools (Emuze replacement) which don't need to carry this information within themselves (as this way they could match UZE files with their EEPROM blocks). To support existing games, such a tool should still add the EEPROM Block ID Reservation List to itself to be able to match UZEs having the EEPROM block ID field zero.

If the Save state file size is nonzero, the game should be accompanyed by a SAV file (example: GAME.UZE => GAME.SAV) of the appropriate size. The clear state of this file is all zeros. This can be used to reconstruct or clear these files in need by a tool.

Further stuff may be added to the remaining 17 bytes later. I don't think games would demand anything beyond these, all what may come are information fields if there was any software which could use them (we already have more info fields than our tools' capabilities, the 16x16 icon and the game description I think was never used).
Artcfox wrote: Wed Apr 18, 2018 12:08 pm Having a standalone utility (a UZE file) that contains the FS_Create_File function sounds like a good idea, especially since it could query that field (if present) from each UZE file on the SD card and offer save file management: create, wipe, copy (say for backing up a save file to allow someone else to play a new game without losing your progress in games that only support one save slot).

If there is enough room in that utility, we might even be able to add a file defragmenter. That is something that I always thought might be fun to create.
I think this is a good idea! For FS_Create_File you will need to implement (or fetch from somewhere) a fairly complete FAT16 / FAT32 filesystem library, with that, defragmentation should also become a possibility (checking for fragmentation is already possible using the bootloader API). This tool could replace Emuze.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: The UZE file format

Post by nicksen782 »

FS_Create_File ? We are starting to talk about a file manager now. Maybe a defragmenter even? I would suggest detecting fragmented files, looking for a big enough block of free space and then moving the data there.

I see the data of a game as three parts. The game's executable (.uze file), the game assets (.bin file), and the game save slot (.sav) each having the same base filename but with different extensions.

At game init, a game should make sure that the required external files exist. Responsibility of the programmer to make sure of this.

Perhaps the header of the .uze file tells all? Could the .uze file could contain a list of required file names and provide a warning before flashing to indicate missing files too? I still think the game should check. But it seems like the whole goal here is to make sure you aren't forgetting a file.

I really like the HW feature requirement flags. Super idea. Maybe indicate this in the bootloader somehow? No guesswork on what you need for playing the game. I like the proposed changes to the .uze header. Don't forget Uzenet though!
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: The UZE file format

Post by D3thAdd3r »

Jubatian wrote: Wed Apr 18, 2018 6:20 am Huh, aren't you confusing the SD and the 644's Flash here? Frying the Flash of the 644 isn't that easy, you can't do that even with the new bootloader unless you left your Uzebox running unattended overnight when the "game" decided to start killing it.
No I was going off tangent a bit. I was meaning something like "BootJacker" for partial SPM writes(many games could use this to replace graphics blocks, code would be dangerous/hard), even if there was never a bootloader which supports controlled SPM writes intentionally. I mean something like setup the SPM register in user code, setup a timer, and call the bootloader SPM sequence with the specifics having been determined in user code. The timer ISR fires off at the first cycle of SPM(cpu stalls until SPM complete), pop the (bootloaders)return address, and user code setups up again to repeat the process of writing arbitrarily. It was my understanding there is no way to stop such a thing, nor even to really obfuscate it in the bootloader, since the SPM setup must happen within 4 cycles of the SPM itself.
Post Reply