The UZE file format

Topics on software tools like TileStudio, comments on documentation and tutorials (or the lack of) should go here.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

Huh, lots of stuff happening here!
D3thAdd3r wrote: Sat Apr 14, 2018 6:20 pmSectors for additional game data--what exactly do you have in mind for this?
This is the simpler to answer: Think of a game like Tempest: it has extra data (the webs), however such extra data which is not realistic to be user-made. Also think about something like Shadowgate if it was realized: it is a story as-is, no user made content. In these games there is no practical reason to have more than one file as there is no point in handling the data file separately from the game executable.

So this additional game data feature would serve this demand: You can add extra stuff which is not to be burned in the Flash, but rather you will load it later. SPI RAM games could use this more extensively.
Artcfox wrote: Sat Apr 14, 2018 12:58 pmI think we could have special magic bytes written to the 16x16 icon field and a pointer to the 32x32 animation, and length.
There is no point in adding "magic" stuff there when there is plenty of room still available within the 512 byte header. The header can contain such info. However I would only add it once there is any realistic possibility of using it (there is no point in adding information fields if there is nothing which could use them: currently neither bootloader have any realistic possibility of implementing such a feature, nor there is any emulator which would offer a game selector which might).

Then the save states.

First of all, you seem concerned about write safety. I think if the game uses the FS library to write, it doesn't matter what concept it uses, safety is the same. If the game fails to correctly select the area to write into, it will cause big trouble on the card. No matter where it should have been writing (region in UZE file, central save file, individual save file, whatever).

Having a kernel library for such a feature makes things safer as the library itself can be made safe.

Now what are the pros and cons of each?

Generic unregulated own save file(s) for each game
  • (+) Only this method allows for true user made content in a convenient manner for the user.
  • (-) Since we don't want to add full FS support for its complications, the save files must be there and sized appropriately to work. This negates almost all benefits which this method would offer. If the user omits the file or deletes it, the game becomes inoperable, it is so difficult to clear saves (if you don't have a clear save file, you are busted). So you still can not move the game without saves.
  • (-) It would be difficult to make proper kernel support for this.
  • (-) It would be difficult to make a generic save state clear function for this.
Central save file
  • (+) It can have proper kernel support (due to the file's standardized structure)
  • (+) It can have a generic save state clear function (due to the file's standardized structure)
  • (-) Due to the fixed size it has to be large to accommodate demands, although it could be mitigated by standardizing it with an arbitrary capacity which you can set to your needs (the games on the card).
  • (-) Moving saves is difficult, particularly if the user had games on two SD cards, he would need some software assistance to merge saves if he wanted to move them onto one card. This can get bad if he occasionally plays on emulator and occasionally on real hardware: even if he plays different games, he can not easily merge saves.
Save data in the UZE file
  • (+) It can have proper kernel support (due to the area's standardized structure)
  • (+) It can have a generic save state clear function (due to the area's standardized structure)
  • (+) Moving saves is easier as two games won't interfere with each other. You just move the game and the saves get moved with it.
  • (-) The area still has a fixed size, however at least this size belongs to one game and can be set by the game author.
You could also have an own save file in a standardized manner as a fourth option: more files on the card, but at least the new bootloader wouldn't make a fuss about it (you can have any number of files). It is only slightly bad for FAT16 (as it has a fixed root directory). Otherwise it is similar to having the save data in the UZE.

I really don't like the central save file approach. It would work for high-scores as high-scores aren't so much essential for gameplay, but if you started really using the SD, then you could start to have more important data, such as a real save location. The player may really want to be able to move it somewhere else (such as from emulator to real hardware), and the central save file could make this a big mess. So I side with either .UZE file field (less files on the card) or standardized own save file (more files on the card), a generic own save file certainly should be a no-go (without real filesystem functions).

When I mention the UZE file, I say absolutely no to expand it from within the game without any awareness of the filesystem. There should be some new indicators in the UZE header which reserve a region within the file (after the game binary) for the save data, which when empty, is zero. So the area would always exist.
User avatar
D3thAdd3r
Posts: 3219
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: The UZE file format

Post by D3thAdd3r »

I was writing up a point by point list to match your pros/cons, but the majority of them I totally or mostly agreed to. But there are some points I feel are not expressed there.
Jubatian wrote: Sun Apr 15, 2018 4:38 pm The player may really want to be able to move it somewhere else (such as from emulator to real hardware), and the central save file could make this a big mess.
To be sure, in the case where game data is static like Alter Ego(no level editor), I think the UZE packing is the best method by far. The "high score" requirements for that game are 1 byte per episode(just storing the last level completed) and it already requires the SD. In that case, it makes sense to me to not use a whole EEPROM block up, and just write the byte to the SD in a standard file...but which one? The UZE method could work there too, but then the state travels with the rom. That is the real question in my mind, should state be part of the rom, and if so, should it be in all cases? It is hard to know what a clean rom is then, where with the central saves at least we are putting all the dirty laundry in one basket when possible.

Toorum's Quest allows the user to create expansion episodes via the editor, so what to do with that..not sure. Should the rom image carry around levels the user has made...it seems the user probably would prefer this for this game..maybe not. I would probably lean towards packing it in the UZE at this case. In all cases, I don't think any of these as currently described are a complete solution for how to transfer data from emulator to hardware.

The centralized approach was devised when doing some additions I intended to add to Emuze. So the basic part described on the wiki(not the extended chain stuff), would more or less be intended to operate similar to EEPROM(and reserves a swap space for Emuze to work on EEPROM). My idea there would be that you could store/restore different versions of the EEPROM on the SD card, and in most cases the ID system used would allow them to be easily merged if they are small enough. With the centralized file idea it is the same thing, where a utility can easily merge these. I fail to understand how any other solution could offer merging abilities without the use of a utility. Without the centralized file having a generic format, it is impossible for the utility to know the ins and outs of each UZE file....unless a generic save section is defined inside the UZE?
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: The UZE file format

Post by nicksen782 »

I'll provide an example idea I had for saving state.

Legend of Zelda had lots of data in its data file (separate file from the .uze file.) And it is only changed when re-compiled and all pointers to regions of data are updated too so the game only see the new data is isn't affected by the fact it moved. But Zelda would have many state-related things in the game such as which keys were collected, which items obtained, number of rupees, doors that have been unlocked. Originally I was going to come up with a way of getting all of that into 512 bytes so that it would fit in a sector. Then, I would use the data file for the save states. Yes, the first sector of that file would be reserved. When the game is loaded it would read that data and fill the relevant local variables and SPIRAM. When the game state is saved then the local variables and whatever is relevant in SPIRAM would be updated to those reserved 512 bytes. That was all the writing I intended to do with the game. Saving data in the .uze file just seems like you are putting data where it doesn't belong. If your game is already using a file from SD for data then save your data there. The file is required for the game anyway and your game ought to tell the user if the file is missing.

I don't see the point of centralization beyond high scores or maybe wifi settings or something.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

Some games truly would demand different stuff, but let's see how these could work.

If you stored the save data in the UZE, obviously you have no merging problem (across different games): the save is carried in the UZE, so copying it around you copy your saves with it, without affecting other games on the card.

The EEPROM I think works fairly fine just because almost all games use it for high scores: usually those are not that important that you even started pondering on how you could transfer them. However with Alter Ego and similar games which store progress, you might like to transfer, and it is then a problem with the EEPROM too (think of that like you try the game out in the emulator, complete ten levels, then you don't want to start over on the real thing). So I think it is better if saves travel alongside the games they belong to (so on the SD card with whatever method using it).

If you stored in the UZE, the header would receive two new fields: Save area begin address and Save area length (in 512 byte sector units as you must write entire sectors, so for safety this is a sound requirement). If by specs you set that a game must treat zero data as being uninitialized, then you immediately have the generic method for wiping: you can easily wipe individual games, or even all of them on an SD card (like the bootloader you can write a function to find all UZE files, then you can wipe each). So if you want, you can have a full wipe with this to get a "clean" card with games on it.

User made content (like Toorum's Quest's levels) would demand full filesystem support for something convenient (which is quite difficult to do), without that, it will remain somewhat cumbersome, and definitely no kernel support can be realized for this (so you can not make these writes truly safe). I think such things should be left for the game author to decide upon.
Last edited by Jubatian on Sun Apr 15, 2018 7:01 pm, edited 1 time in total.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

nicksen782 wrote: Sun Apr 15, 2018 6:47 pmThe file is required for the game anyway and your game ought to tell the user if the file is missing.
Then why don't set things up so the file can not be missing since it doesn't even exist? :)

That's what I would recommend by the extra data support in the .UZE: things which are integral to the game should go along with the game, not extra files (as it gives no advantage to the user that the game is sitting in two files as neither file is useful in any way on its own without the other).
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: The UZE file format

Post by Artcfox »

If we define header extensions so things like the 32x32 animated icon extension can be implemented without clashing with level data, I think that it would be okay storing extra level data at the end of the uze file, but I really don't think save data belongs there. If someone like Adafruit is making a ROM pack to distribute with their kits, they'll probably test out the ROMs they include to ensure a good experience, and then they might inadvertently include ROMs that don't start at the beginning of their games, because they tested them out first using an emulator. I don't like the idea that I might have to be extra careful when giving someone new to the Uzebox a "pristine" set of ROMs, I want to just be able to copy them to their SD card and let them do the same when they want to spread the Uzebox love to someone else.

If people insist on including static level data at the end of their uze file, I think that it should remain read only. Especially since the new bootloader doesn't count non-uze files toward the maximum file limitation, and FAT32 does not have any limitations.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

Artcfox wrote: Sun Apr 15, 2018 7:43 pm(...) and then they might inadvertently include ROMs that don't start at the beginning of their games, because they tested them out first using an emulator.
This is a valid concern, however I don't think it has any sane solution without full filesystem support (the ability to create new files). Even then, after testing, they could still accidentally leave the save files there.

The problem is that even if the save sits in an external file, you have to have that file properly prepared for the game (as it can only overwite it, and can not modify its size, nor can create a new file). Those files could be accidentally left in a non-blank state (whatever this means for the game), same result.

To avoid this, a central save file would be required which is more likely to be noticed, however that has the aforementioned other problems (very inconvenient for the user as it is difficult to move saves of specific games from SD card to SD card or to/from emulator).

I would rather side with the aforementioned mass-wipe tool, which could mass-clean save states for a card.

(By the way, extra game data in the UZE is of course only an option. I would like to have it as I see it more elegant to have all stuff integral to a game within one file, but it does no harm if you decide to rather still use an external file even if the UZE format was expanded to support extra data)
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: The UZE file format

Post by CunningFellow »

Add the new field to the Uze format which is

"First Sector of User Data"

Rather than assume that user data starts straight after the ROM. That is future extendable.


ALSO - I would advocate writing is only allowed to be done via the bootloader. Be it a central file, a separate file or at the end of the UZE file. Make the bootloader the only thing allowed to write. Then build in safety checks so that a write can only happen within the bounds of a specified filename.
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: The UZE file format

Post by Jubatian »

CunningFellow wrote: Sun Apr 15, 2018 8:21 pmAdd the new field to the Uze format which is ...
That's how I plan to add these: The point of the discussion here is especially whether such should be added to the header so these things could receive standardized support.
CunningFellow wrote: Sun Apr 15, 2018 8:21 pmALSO - I would advocate writing is only allowed to be done via the bootloader. (...)
Impossible. If you wanted to mention the bootloader library's FS_Write_Sector() routine, then more possible. The bootloader itself has no write (it didn't fit there), the bootloader library uses the bootloader API (if it is present) proper to add write support over it.
CunningFellow wrote: Sun Apr 15, 2018 8:21 pmThen build in safety checks so that a write can only happen within the bounds of a specified filename.
A new library for writing stuff. That would be the point in standardizing a save area within the UZE, or a standardized central save file: then a library can be made for it which ensures writes can only happen into the correct location. But as of now the limitation (of either solution) is that only a fixed range can be specified which exists somewhere in some file (no filesystem support for creating / expanding). If a game needed something more elaborate, it would still have to do its own.
User avatar
D3thAdd3r
Posts: 3219
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: The UZE file format

Post by D3thAdd3r »

One thing I didn't see discussed, how do I build an UZE with embedded extra data right now?
Post Reply