Bootloader API library with SDHC and FAT32

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Artcfox »

Everything is in sectors, and it reads an entite sector at a time into a 512 byte buffer. You can then access any part of that 512 byte buffer as an array to grab stuff at whatever byte offset you might need within that sector.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by nicksen782 »

I tried to dump a text file and was able to but there did not appear to be a way to detect EOF. It was like the last sector would repeat several times until res came back something other than a 0.

However, I took a streaming music binary and read it from the SD file. The file size was an even multiple of 512. I read it into SPIRAM and had no problem streaming music. Of course, bulk transfer is easy. Mostly because I knew exactly how many bytes to read. That's really the problem. But it isn't a new problem.

After working with it for a while I figured it out. Thanks, Artcfox!
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Artcfox »

nicksen782 wrote: Sat Apr 21, 2018 2:27 am I tried to dump a text file and was able to but there did not appear to be a way to detect EOF. It was like the last sector would repeat several times until res came back something other than a 0.

However, I took a streaming music binary and read it from the SD file. The file size was an even multiple of 512. I read it into SPIRAM and had no problem streaming music. Of course, bulk transfer is easy. Mostly because I knew exactly how many bytes to read. That's really the problem. But it isn't a new problem.

After working with it for a while I figured it out. Thanks, Artcfox!
Glad I could help.

So for binary files, I always include a header that has the length of the file, either in sectors or bytes, whichever is appropriate. Then by reading the first sector, you'll know where the end of the file is.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by nicksen782 »

Yes, a good idea! I previously was using offsets within the file. I have a C program that puts together the assets file and it creates a file with a list of #define to include.

Nice that the new SD library handles things like the Chip Select. It reads one sector. You can request another sector afterwards and then read it so I can still read all my data.

I'm really looking forward to the UCC2018. I've been quite busy! I know you have been too.
User avatar
Jubatian
Posts: 1563
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Jubatian »

A little late, just a few details.

There is an example in Mode 748 which deals with non-sector boundary data, it is the row mode 4 scroller. It loads a 4bpp bitmap into the SPI RAM, but while loading, it sorts the input to different regions to make scrolling possible.

And yes, the Bootloader API accesses whole sectors, a simple interface which was necessary for the bootloader. Calculating CRC wouldn't make it strictly necessary, but it would make the interface complex.

The end of the file can not be detected. The simple no fragmentation fallback (and what the original bootloader and older SD libraries did) has no way to detect it during reading, the file size would have to be read in advance and stored somewhere. So you have to know how large the file is. If you want text files without making a FAT16/32 aware filesize query function, you may use some end of text marker.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by nicksen782 »

I have it working.

Question, I have music at the beginning of my file and then afterwards I have some graphics stuff. In the past I would jump to the correct sector boundary (offset within file % 512). Something like this:

Code: Select all

	// Reserve enough ram tiles to prevent sprite blitting errors.
	SetUserRamTilesCount(((bytesToRead+1)/64)+1+1);

	// Read up to the start sector of the requested offset.
	sdCardCueSectorAddress(sectorStart+(offset / 512));

	// Skip ahead until at the correct byte.(avoid skipping 0 bytes.)
	if(offset % 512) { sdCardSkipBytes(offset % 512); }

	// Read the data into the specified destination (pointer.)
	sdCardDirectRead(dest, bytesToRead, 0, 0);
Basically the same thing right? Get to the closest sector to the data you want within the file and then skip bytes until you reach the data? I could probably just arrange data to be on 512 byte boundaries.

So, lets say that I needed to get to the bytes directly after the music and the music is 9728 bytes long. This is exactly 19 sectors. Do I just do "FS_Get_Sector(&sd_struct)" in a for loop for 19 iterations? Is there a better way?

Also, potential bug (or I'm messing something up.) Function_FS_Get_Pos just freezes the program. How is it used?
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Artcfox »

I would align it on a sector boundary.

Then you can do:

Code: Select all

SDC_Read_Sector(&sd_struct, FS_Get_Sector(&sd_struct) + 19);
to skip ahead 19 sectors (assuming your file is not fragmented).

If the file might be fragmented, and you don't want to use the low level routines, then you can probably just call FS_Next_Sector in a loop 19 times, and then use FS_Read_Sector to actually read the data, which should update the current position inside the sd_struct to point to the next sector.
User avatar
Jubatian
Posts: 1563
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Jubatian »

nicksen782 wrote: Mon Apr 23, 2018 9:27 pmAlso, potential bug (or I'm messing something up.) Function_FS_Get_Pos just freezes the program. How is it used?
These functions should be used for storing away and retrieving compact file positions (4 bytes each), so you can do random access fast once you walk through the necessary part of a file with FS_Next_Sector.

I remember testing them, however as of now the code of FS_Get_Pos looks jumbled up, possibly something went south between the tests and the actual commit. Maybe I accidentally committed a different file than which I intended to (of course this case the correct file is certainly missing as I usually wipe work files after a commit to prevent later accidental regressions). I will fix this later today.
User avatar
nicksen782
Posts: 714
Joined: Wed Feb 01, 2012 8:23 pm
Location: Detroit, United States
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by nicksen782 »

Artcfox wrote: Tue Apr 24, 2018 1:59 am I would align it on a sector boundary.

Code: Select all

SDC_Read_Sector(&sd_struct, FS_Get_Sector(&sd_struct) + 19);
SDC_Read_Sector takes 2 arguments. FS_Read_Sector takes one and requires FS_Next_Sector to queue up the next sector. So, if I kept track of the sector myself as you suggested in your example, what is the difference between SDC_Read_Sector and FS_Read_Sector? What about fragmentation?

Edit: I may have just answered my own question. FS_Next_Sector supports fragmentation while SDC_Read_Sector does not support fragmentation.

So, I could do the FS_Next_Sector several times to get to the data that I want. I'm not doing real time time-sensitive reads (like Tornado 2000) and I expect to have some loading animation occur during so it won't seem too slow anyway. With SD Simple streaming I filled the entire 128k SPIRAM in under 1 second and even faster if I used SetRenderingParameters to reduce down to one tile (which had my loading progress bar.) Would the time be that much different with the new bootlib?

Is there a better way other than doing FS_Next_Sector repeatedly from the start of the file? My data file may be large enough to be concerned with speed for data towards the end of the file.
User avatar
Jubatian
Posts: 1563
Joined: Thu Oct 01, 2015 9:44 pm
Location: Hungary
Contact:

Re: Bootloader API library with SDHC and FAT32

Post by Jubatian »

I fixed the bug. It now seems to work fine, you can use FS_Get_Pos() and FS_Set_Pos() as described in their documentations.

Of course if you just need a one-shot fill, walking through the file with FS_Next_Sector() and FS_Read_Sector() is fine. The FS_Get_Pos() and FS_Set_Pos() functions are mostly intended to allow for to reload stuff later without making a seek from the beginning of the file necessary.
nicksen782 wrote: Tue Apr 24, 2018 2:24 pmEdit: I may have just answered my own question. FS_Next_Sector supports fragmentation while SDC_Read_Sector does not support fragmentation.
SDC_Read_Sector() is just a lower-level routine. FS_Read_Sector() uses it. The FS_Next_Sector() routine provides the sector information for reads as it walks through the cluster chain in the FAT, the actual position information is stored within the sdc_struct_t structure (actual format is unspecified, depends on the bootloader or fallback library used, FS_Get_Pos() and FS_Set_Pos() are provided to work with this info in a universal manner).
Post Reply