Bootloader API library with SDHC and FAT32

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
Post Reply
User avatar
Jubatian
Posts: 1564
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Bootloader API library with SDHC and FAT32

Post by Jubatian »

I finalized and merged in an initial version of the bootloader API library.

Important: The library does not require a bootloader to be present for using the SD card. If there is no suitable (V.5.x.x, brown interface) bootloader, it reverts to an MMC + FAT16 interface. This is the same like included in the V.0.4.5 bootloader (blue interface), so games using this library should work fine everywhere (including old bootloader Uzeboxes and the Uzem emulator).

I added a demo (BootDemo) alongside, which is also useful for checking SD cards (program the demo, remove the card, insert another, hit reset, see what it says).

The library normally aims to be safe, using the card with CRC enabled at slower speeds. This is essential if you fixed your Uzebox with the V.5.x.x bootloader to get over SD card instability. You can turn off these to make fast access routines such as for streaming.

When doing that, ideally you could design in such a manner that only graphic streaming is unprotected, if you need any data which is essential, you may turn on CRC again and use the library's routines to fetch those.

Fragmentation support is possible alongside some streaming if you assume a minimal cluster size (if the filesystem has 8Kbyte clusters, then 8Kbyte blocks of data are contiguous). You can also build checks for fragmentation as needed if you don't want your game just failing when the data is fragmented.

SD writing is not included. I will later add support for it, for now I thought it critical to have at least reading supported proper.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by nicksen782 »

Awesome. I'm looking through the demo code and I see a 512 buffer being created for holding a sector. However, for quite some time I have been using reserved ram tiles or even vram to temporarily hold the data. Of course, it is probably different with the SD Simple library that just streams. No real knowledge of the FS there other than being able to find a file.

bootlib.h is nicely commented. I was looking for the typedef of the sdc_struct_t . I see that the bufp member is a pointer. Also, you can set bufp to an array (such as the local 512 byte one in the demo.) But, what if you reserved 8 (64 byte) ram tiles and then used &ram_tiles[0] instead?

So, store the sprite X and Y, then turn off sprite processing, aim bufp at ram_tiles, do the read into wherever and then turn sprite processing back on? Can that be done?
User avatar
Jubatian
Posts: 1564
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Jubatian »

nicksen782 wrote: Wed Jan 03, 2018 5:56 pmbootlib.h is nicely commented. I was looking for the typedef of the sdc_struct_t . I see that the bufp member is a pointer. Also, you can set bufp to an array (such as the local 512 byte one in the demo.) But, what if you reserved 8 (64 byte) ram tiles and then used &ram_tiles[0] instead?

So, store the sprite X and Y, then turn off sprite processing, aim bufp at ram_tiles, do the read into wherever and then turn sprite processing back on? Can that be done?
Yes! You can point it to any area free of writes, for example in some video modes you could even use the VRAM after turning off display by whatever means appropriate (such as by SetRenderingParameters(), setting the height zero). You only have to keep in mind that when you are using the library routines, that 512 byte area you provide needs to be available (noone trying to write in it for example within the video frame interrupt).
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by nicksen782 »

Great! So, the SD Simple library is just a streamer and therefore doesn't need much of a buffer. But your version does?

Doesn't really matter since you only need that 512 bytes for a split second anyway. In fact, if you render only the top vram line (for something like "Loading") then you gain quite a bit of speed in the SD read. This I've noticed myself.

So, reserve 8 ram tiles, do the read and put the data where it needs to go, then return those returned ram tiles? And if using vram then reduced the number of rendered lines and then back again?

This makes sense. I think I get it.

But, I have to ask the following question.
Why use it over the SD Simple? Is it for flash usage? ram usage? More simple? Benefits to users with the new bootloader? An SD library that works with or without the help of the new bootloader?
User avatar
Jubatian
Posts: 1564
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Jubatian »

nicksen782 wrote: Wed Jan 03, 2018 7:19 pmBut, I have to ask the following question.
Why use it over the SD Simple? Is it for flash usage? ram usage? More simple? Benefits to users with the new bootloader? An SD library that works with or without the help of the new bootloader?
It has the following advantages:
  • SDHC, FAT32 and fragmentation support.
  • Robustness, which is critical when you load anything not merely displayed or played through the speakers.
SD Simple by its concept could only add SDHC and FAT32 support, but wouldn't be able to be robust. Robustness is very critical for anything beyond streamed video or audio. It seems like it is rather common that at max SPI speed Uzeboxes are not necessarily reliable accessing the SD cards. If you want to load for example level data between two screens, then better be sure to really have that data what you need than getting it fast (otherwise the gameplay will suffer or the game might even crash or make the player stuck).

SD Simple I think should be converted to a streaming library which could be used on top of this one. So when you want to stream some data, you would use SD Simple's functions, otherwise rather use this library to make sure you are loading exactly what you need.

To achieve robustness, it is necessary to read full SD sectors so the CRC can be checked. So that way of access not requiring a sector buffer that SD Simple provides is inherently unchecked, and could lead to failures on "weaker" Uzeboxes or SD cards. This is of course OK when you are streaming (a click in audio or a bit of noise for one frame on the screen).
User avatar
Jubatian
Posts: 1564
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Jubatian »

I added documentation for the library on the Wiki:

Bootloader API Functions

I did it a little broader than what is included in the header file, hopefully giving some further ideas on how to use it.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Artcfox »

Jubatian wrote: Thu Jan 04, 2018 9:26 pm I added documentation for the library on the Wiki:

Bootloader API Functions

I did it a little broader than what is included in the header file, hopefully giving some further ideas on how to use it.
I read through them all, and it looks great. The only thing I wish was added was a wiki entry for the sds structure. Sure I can look in the code to see it, but it might be nice to also put it on the wiki, because it would help support the documentation. Nice work!
User avatar
Jubatian
Posts: 1564
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Jubatian »

Artcfox wrote: Fri Jan 05, 2018 4:06 am(...) The only thing I wish was added was a wiki entry for the sds structure. (...)
No, you don't want to look at it! :lol: That structure should normally not be meddled with directly apart from assigning a 512 byte buffer to its "bufp" field which is described where appropriate.

I found the area where I should be adding such, although for me it looks a bit messy... Maybe it all would really benefit from a bit of renovation.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Artcfox »

:P
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: Bootloader API library with SDHC and FAT32

Post by CunningFellow »

I would say that now there is no reason at all to use SDSimple for FAT/Directory/File finding things.

Use the new API.

The only use SDSimple could still have is to read data hunks as fast as possible into a buffer if CRC does not matter. If you need to get the data at 17 clocks per byte then the read parts of SDSimple have had much effort applied to make then as fast as possible.

In fact when I get time I am going to update T2K to use the new API for finding the movie file so I can support SDHC as well as SD. After the file is found the only difference between SDHC and SD should be the byte vs sector address in the "read multi" command.
Post Reply