Uzebox Data Manager - For managing your local Uzebox library

Topics on software tools like TileStudio, comments on documentation and tutorials (or the lack of) should go here.
Post Reply
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Uzebox Data Manager - For managing your local Uzebox library

Post by nicksen782 »

Discussion started here:
http://uzebox.org/forums/viewtopic.php? ... 210#p30501

We have WiFi through the ESP and we have a games database. What if we could use both of those to create an Uzebox "game" that would serve to help you download new titles and keep existing titles up to date... all from your actual Uzebox console?

Of course, this will require the ESP to be installed into your Uzebox. That comes with the SPIRAM built on the same PCB so if you have one then you likely have both.

Goals:
* Download a master game manifest from a remote server. Save that to SPIRAM since it will likely be larger than the RAM available.
* Option to hash all the files on your SD card and save the hashes to a text file.
* The game manifest would have a hash of every game file. That hash could be used to determine if your games and files are not the official version. This can flag games as needing an update.
* Download individual games to your Uzebox.
* Download ALL games to your Uzebox.
* Maybe something could be done to get Tornado 2000 to fit. The download itself is only around 50 megs but uncompressed it is half a gig. Perhaps there is a minimal unzip utility out there that could be used. If so then I would use it for all the game downloads to save download time.
* WiFi setup: Use this to configure the ESP for your WiFi connection. I do not know how this is done now but I would want to create this anyway.
* Using minimal ram tiles and sprites and a reduced height vram I got a base program using around 900 bytes of ram. Ram will be essential so I do not intend to add much to the visual appeal until the main functions of the program have been tested to work. Whatever remains can go to graphics/sound.
* This program will not be about sharing Uzebox files between Uzebox consoles peer-to-peer. This will be a client-server thing.
* I do want the ability to easily download games that are NOT in the database as well like a program you are working on. Does your game have an update? Great, use this tool to update it right away on the Uzebox and test it out.
* What about an SD card speed test?

I can figure out the above for the most part other than the ESP which I have never used on the Uzebox before. I'm going to take a look through the Tank-Fu code to see if I can figure out how L4rry did this.

We are about to have an Uzebox portable! This tool would be a great addition to Uzebox and I would use it often. Imagine compiling your game and then copying the file to a remote server from within your Makefile. Maybe even a local network share? Then, to load it into your Uzebox you would just run this application. It could be a way to more easily develop on the hardware without swapping around SD cards.

I'm open to suggestions. I'm just trying to get the WiFi basics at this point but then the real fun begins!
User avatar
L4rry
Posts: 242
Joined: Sun Dec 28, 2014 7:19 am
Location: Cape Town, South Africa

Re: Uzebox Data Manager - For managing your local Uzebox library

Post by L4rry »

Great idea! This will be super handy to have. You should have seen the daily SD card swap circus needed to debug networking for Tank Fu :lol:
nicksen782 wrote: Sat Jan 26, 2019 12:06 am I can figure out the above for the most part other than the ESP which I have never used on the Uzebox before. I'm going to take a look through the Tank-Fu code to see if I can figure out how L4rry did this.
Be sure to have a look at the Uzenet Demo first. I would recommend starting there as it actually implements Http calls (see HttpGet method defined in uzenet.c). Also, because of it's console support, it's a convenient place to debug your ideas. All the ESP AT command combinations for Tank Fu was first tested by hacking the UzenetDemo code.

Tank Fu has some decent routines for initializing your ESP module without knowing it's current baud rate as well as resetting the module. See the initWifi() function in 'uzenet.h'
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Uzebox Data Manager - For managing your local Uzebox library

Post by Jubatian »

Good ideas, but I feel this could grow into an enormous effort, and I see some feasibility problems too.

Mostly there would be that it wouldn't make it a lot easier to debug on real hardware as you would still have to swith between applications on the Uzebox itself. The workflow would be about as follows:
  • Request Game Loader to load the Wifi game manager.
  • Fetch your game by the Wifi game manager. How convenient this can get depends on the UI of the game manager, so it is up to you.
  • Back to Game Loader, select the downloaded game to load. This step may be skipped with the new bootloader which has an interface to burn a new game.
  • Test game.
  • Back to the beginning.
With the new Game Loader, if you used its interface to burn the game, this may work okay as the Game Loader remembers the last programmed game, this case the Wifi game manager. So it would be easy to select it (a request by interface wouldn't modify this stored last programmed game). With the old bootloader the whole thing would be such an arduous chore that it would be easier to just keep an empty SD card around and use that for development.

A problem with the approach is that for every test of your game, you age the Flash twice, and I think real HW debugging is already one of the most costly activity in this regard.

A critical feasibility problem is the writing to the SD card.
  • The bootloader interface library does contain write support, but only into existing space in existing files, in accordance with the minimalist filesystem implementation in the bootloader. It is not suitable for this task. You will need to add PetitFatFs or something alike having full Fat16/Fat32 support on top of the SDC routines in the bootloader interface library (so you step one level lower to allow for a more complete filesystem support).
  • The new bootloader works fine with a fragmented filesystem, you can use an ordinary approach to access the FS, and it will work fine. However a couple of games including Tornado 2000 don't. You would possibly have to defragment, that's something you probably wouldn't be able to get a third party library for.
  • Most existing games using the SD card can't work with SDHC (including Tornado 2000). This is just something which can not be solved at all (without modifying the games themselves).
Not that it is not a good idea, it would be quite nice to have, but to solve these adequately would demand thought and effort, probably a lot.

By the way, what would be the significance of using Mode 3 with this tool? It is probably more text intensive than graphics for all the configuration requirements and selections, so it feels like one of the higher resolution, more text-oriented modes would possibly do better for it. With Mode 3, you would only just have enough tiles to print out a game's name in the .uze file (30 characters).
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Uzebox Data Manager - For managing your local Uzebox library

Post by nicksen782 »

@ L4rry: Yes, I noticed in the Tank-Fu code that there were not any HTTP calls. Still was good to read through it! Where can I find the list of ESP AT commands? Great advice, thank you. I'll look through the Uzenet demo. That appears to be the best starting point.

@ Jubatian: You are right. The initial proposed scope was rather large. I swung a wide net and I really appreciate your feedback and concerns. The biggest concerns that I am seeing are the increased number of flash writes and the inability to write a new file to the SD card. You say this is a limitation of the bootloader/fs library. I don't think this is a limitation with Petit FS (which I have used previously.) I would need to use that full library. I'm not worried about flash usage because I am going to start with a very minimal graphics UI at first. I'm more concerned with RAM.

Yes, I was considering other video modes. I first looked to Mode 5 since it is like Mode 1 but with 8-bit vram indexes. But, the tiles were 6x8 and I didn't know if you could adjust them to 8x8 like you can in Mode 1. At this point I decided to use Mode 3 with minimal sprites and ram tiles (4 each) for something like a cursor. I am not opposed to other video modes as long as their complexity does not get in my way of actually getting something on the screen fast. Again, I'm thinking simple UI until the feature set is actually set and working. It would be NICE if I could get 40 to 80 columns though so I don't have to worry as much about type-fitting the text on the screen.

Perhaps using it to develop direct on hardware then isn't such a good idea when you can just keep a spare SD card with your game on it. However, I still like the ability to pull in a remote game (at any stage of development) and still would like to implement that. But the "Swap Circus" would need to continue to be a part of development when developing on the hardware. That's fine. I mostly develop on the online emulator anyway. CUzeBox doesn't support the ESP at this time, right? Could it? ;)

Revised scope:
How about the tool be used for updating the existing local library in whole or in part, loading non-official titles, WiFi configuration, and SD test (or other utilities.) It may sound silly but with 256k of SPIRAM we could actually build a legit de-fragmenter but in a simple way. All reads/writes must be 512 bytes (sector) and no file is likely to be above 256k (we will just have to check for that.) Copy the file to SPIRAM in full, then write it back as a new file upon determining that enough contiguous space exists. This is a NICE TO HAVE... not a need. Focus should be on updating the local game library on the SD card.

Any more ideas? Thanks again. Let's get the scope defined now so that we avoid much creep later :)
User avatar
Jubatian
Posts: 1560
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Uzebox Data Manager - For managing your local Uzebox library

Post by Jubatian »

nicksen782 wrote: Sat Jan 26, 2019 4:02 pm Again, I'm thinking simple UI until the feature set is actually set and working. It would be NICE if I could get 40 to 80 columns though so I don't have to worry as much about type-fitting the text on the screen.
Mode 9/90 and Modes 40/41/42 are quite simple to use, the latter even have built-in fonts to help you get started, the envisioned tool I think could do pretty well with one of these, to have an ASCII or PETSCII (C64) look and feel. Of course also depends on the RAM budget, but with the SPI RAM assumed to be there, I wouldn't worry much about that part. This doesn't need to be 60FPS, scanning the SPI RAM for information slowly isn't a big deal here. So I think it was feasible to afford a more RAM hungry mode. Modes 40/41/42 are unique in that regard that they offer attributes, and you can colour the text and its background for added information (the modes vary in what sort of attribute mode they provide), a flexibility no other video modes are offering.
nicksen782 wrote: Sat Jan 26, 2019 4:02 pmCUzeBox doesn't support the ESP at this time, right? Could it? ;)
Maybe Lee (D3thAdd3r) has something working somewhere in this regard, he attempted an implementation at some point, I don't know where he got with that. To make an official implementation, however there are some significant blockers to cope with, such as how the related configuration would be stored, and how the implementation could coexist with the UART hack CunningFellow's RLE mode demands (the proper resolution of which would be an in-depth study of the UART's timing, lots of real HW experimenting involved). I occasionally asked CunningFellow whether it was possible to do that differently anyhow, but it seems like no, that's the only way to get that RLE mode. So for now the UART hack is in the master.

Personal issues: new country, new job, and there, a project tits-up due to some requirements creep, I am pretty much drained. I am just a lot more interested in trying to get that beast of video mode running when I have the mind to do any progress. I am just tired of doing research, more research, rummaging through datasheets and manuals to get something done (which is a significant part of my job these times), and these (ESP for CUzeBox or resolving the UART timing) would involve a lot of that.

The SPI RAM is still only 128KBytes ;) - but that should still be more than enough!

Yes, the defragmenter is definitely possible, I just emphasise on it as it is a significant chunk of work to be done. For .uze files copying them into the SPI RAM and writing them back is OK, but there is Tempest and other potential large data files where such an approach wouldn't work (actually, with the new bootloader, you wouldn't ever need to defrag the .uze files themselves as they are only read by the bootloader, which supports fragmentation proper, you "only" need to have the data files of certain games unfragmented).
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Uzebox Data Manager - For managing your local Uzebox library

Post by nicksen782 »

128k... opps! Right!

Too bad about the CUzeBox ESP support. That just means that I'll need to do most of my testing on the hardware. No big deal, I can manage. As far as a config goes though I can read/write the Emscripten filesystem at any time and update files. This is actually how I can keep the EEPROM state between game loads since I read it out of Emscripten and put it in local storage whenever a game unloads. I load it back when a game loads. There wouldn't be much to stop me from writing a config file into the Emscripten instance either before starting it or even afterwards. I could even edit data on the fly if I wanted to (and I might for future debugging in a game.) I actually considered using a config file for gamepad support (since CUzeBox supports that) but I decided not to. Actually, each gamepad button that is pressed is just a key sent to CUzeBox which accepts it as keyboard input.

A defragmenter does sound good. Or, even more simple yet, a fragmentation DETECTOR. This could indicate to the user that they may have a potential problem.

So, right now I'm having trouble with the Uzenet module (I have 2 built) which seem to be crashing the system upon being spoken to. Do I need a firmware update or something?

Honestly, the back-end web server portion of this is fundamentally done since I can just query for a complete list of games and do hashes. I need to get the Uzebox to download that file and parse it. But, my Uzenet modules are not working.
User avatar
D3thAdd3r
Posts: 3175
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Uzebox Data Manager - For managing your local Uzebox library

Post by D3thAdd3r »

I like the idea and you have a significant part done on the backend web part already. In my view, it would be useful to have exactly 1 "perfect rom set" like what you have for the web based emulator. Then just wipe the directory and write everything new sector by sector. Could be a better way to do it. When an existing rom changes size in this solution, there would be redownloading of everything which is not ideal. The simplicity would seem to make it more realistic to complete quickly.

You should be able to get about 8-10KBps transfer rate off the net which is a normal .uze file in < 10 seconds. So maybe 10-15 minutes to get all the roms written on a blank card. Except data files like T2K, AE, TQ, etc. are some megabytes much larger than the actual complete rom set. T2K is about the only thing that necessitates compression, unless one doesn't mind leaving it overnight. There must be a way to do full blown LZW style on Uzebox, but it would be a lot of work I'm sure.

There is some ESP emulation stuff I have not looked at in a long time. I developed game tests with it(the older much slower Uzem version), but for speed and realistic timing(for which totally realistic timing should be possible with this new direction) comes down to the UART emulation. I would like to get back to this and try to fix this part, then it is a matter of perfecting the ESP core. I wish I had time to do it, as I keep dreaming about doing the racing game which needs ESP support for multiplayer/remote AI.
Post Reply