Evolution of the SD card API

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
Post Reply
User avatar
uze6666
Site Admin
Posts: 4801
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Evolution of the SD card API

Post by uze6666 »

It's been discussed on many topics lately, let's discuss here the ideas about the SD interface.

We all agree it needs some sort of overhaul. We need to unify everything in something more simple and coherent. Here's how it started:
  • It started with mmc.s/fat.c base for things like Uzeamp. Though I always found it annoying it only supported reading and no supports for fragmentation.
  • I added some primitive streaming code for Uzeamp
  • I later added FatFS for those needing full a full FAT API and petitFatFs for a lightweight version. Though both uses there own mmc low level lib.
Obviously I think having FatFS and PetitFatFs (fatfs) is important since they support read/write, fragmentation and SDHC and FAT32. More over they are supported library and they have good documentation. That said, it comes to the expense of more overhead and there's no functions for streaming.

Some ideas and comments:
  • Have only one low level lib (the mmc thing) at the kernel level. Adjust fatfs to use it or use theirs.
  • Use fatfs for all FAT functions to locate files, search directories, write files, etc
  • Develop new Uzebox APIs for streaming and other SD stuff that would uses fatfs structures. The idea is to not reinvent the wheel for the FAT stuff and have optimized functions at the same time for streaming.
  • Should we consider SDHC at this point?
  • Those new APIs will probably not support fragmentation for speed reasons. We should have a function that checks the files for fragmentation and warn the user. Eh, that or defragment the file when the game starts! 8-)
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: Evolution of the SD card API

Post by CunningFellow »

My slowly evolving version of mmc.s has

Init without a buffer
cue a sector
stop transmission
skip n bytes
read a byte
read a word
read a long

The three read functions take into account the 512 byte sector and will read across them just fine (with a pause / delay of course). So they are perfect for the streaming functionality. This uses 2 bytes of RAM for a 0..511 counter.

I use these functions to store the fonts/graphics for the Tornado title/credit screens. I just point at the SD card where I held the data and then read it straight from SD card to VRAM/TileRAM.

About the only thing I can think off the top of my head that would add ease of use would be "Start reading at this BYTE". That is if you want to start reading from a byte that is not on a 512 boundary then it will automatically "skip n bytes" for you.


There is also a C function to find the first sector of a file given its 8.3 file name. This file has to be in the root directory, but now the read_byte command works over a boundary it no longer needs to be one of the first 15 files. It takes 352 bytes of flash for that one C function. But I am sure I could knock it down to half that in ASM.
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: Evolution of the SD card API

Post by CunningFellow »

OH - and a command to read n bytes straight to memory with an optional span

mmcReadNBytes(uint8_t *Destination, uint16_t numberOfBytes, uint8_t span)

A 16 bit destination of where in RAM to write the bytes straight too
A 16 bit counter of how many bytes to write
An 8 bit span so you can automatically fill either ROWS or COLUMNS of VRAM
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Evolution of the SD card API

Post by D3thAdd3r »

CunningFellow wrote:It takes 352 bytes of flash for that one C function. But I am sure I could knock it down to half that in ASM.
Sweet, do you have any idea what kind of flash foot print would be required to find the start of a file and have non-streamed blocking reads to ram and nothing more? If memory serves PetitFS has about a 2.5k+ flash footprint just for what I needed in Alter Ego. Just the basics of mount() and read() were used. The other suggestions seem very useful as well.

How many cycles can we expect to seek and read a 512 byte sector on an average card? It seems we are still talking about blocking functions that, hopefully, return quickly. In my mind the problem with streaming in a complete/heavy game is that certain frames, such as when loading a new horizontal game field stripe, could take significantly longer than other frames where game logic might not run at all(because SD was blocking). It would be very nice to have some read function that took a command, kept a state, and you called each frame and passed the number of cycles it's allowed to take. You could poll it's state each frame to know when it's done. It finishes the read when it gets a chance, but wont cause hiccups by missing vsync on the game logic. Even a fast scrolling game should not need to load a "stripe" more than once every few frames.

The same thing applies to music. I ran PetitFS and a quite small circular buffer easily handled the music I threw at it. I couldn't use it in a game because it keeps locking it up, but I didn't ever have a real need to keep the buffer full every frame, just "before too long". That said, attempting to make a finite state machine out of the pf_read() proved non-trivial to say the least. Perhaps that should be handled at the lower level.
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: Evolution of the SD card API

Post by CunningFellow »

In tornado 2000 the whole code to init the SD card, read stuff and find the first sector of the 8.3 named file is 646 bytes.

I am sure I could save at least 100 bytes from that.

I don't know about an "average" card but in tornado 2000 these are the maximum that is allowed before it fails

900 clocks between stop transmission and read_multi

23000 clocks between read_multi and the data being ready (16 scanlines)

100 clocks between 512 byte sectors

The Kingston, Sandisk and Patriot cards I have here all do less than 500, 3000 and 100 respectivly.

Neither the 500 clocks between stop/cue or the 3000 clocks between cue/ready are blocking.

The 100 clocks for inter sector boundary IS blocking AND THERE IS NO WAY I can see to change that fact unless it can use interrupts.

At the moment my getChar() function that is C callable IS blocking (for 30ish clocks) because I didn't need it to not be. I will make it an inline_ASM macro that remembers it's state to make it be able to return a byte in as few as clocks as possible. (I think 6 clocks as long as you have left 18 clocks between requesting bytes)

Reading a whole 512 byte sector would take 13000 to 14000 clocks.

That is where the streaming functions come into their own. You can always read FWD from where you are now for little cost.

If you extended JHHowards idea with not only having row major and column major versions of the map but had

row major travelling down
row major travelling up
column major travelling left
column major travelling right

then you would only ever have to stop/cue/seek when changing scroll directions.
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: Evolution of the SD card API

Post by CunningFellow »

OK - changed the hello world demo to read a text file from SD card.

Add 814 bytes to the program size.

Edit: scratch that. It made it 708 bytes longer. I had left in some other test stuff.

I think I may be able to get that down to somewhere between 500 and 600 bytes for being able to find files on SD card and read them on the fly.
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: Evolution of the SD card API

Post by CunningFellow »

Still not sure about the git thing.

I think I have added the modified tutorial with add SD loading to

https://github.com/andrewm1973/uzebox

under demos/tutorialSD

As I said - I am sure I could get that down to 1/2 a K of flash for reading the SD card though. Fair bit of saving over petiteFS but you only have access to the root directory and only on a Fat16 2G or less SD card.
User avatar
uze6666
Site Admin
Posts: 4801
Joined: Tue Aug 12, 2008 9:13 pm
Location: Montreal, Canada
Contact:

Re: Evolution of the SD card API

Post by uze6666 »

That's a lot of nice functions you got there. I'm all for them enhancing or replacing the base kernel SD API. Are they backward compatible? Do that would be the base api for speed, size and steaming and FatFs for the full shebang.Things is I'm wondering if we could not make thing so both could be used in the same project.

I'll look in more details on your git commit, but already I can see there's a bunch of files that should not be committed. Basically all the compiled stuff and derived artifacts like *.o , *.o.d, *.hex, *.lss, that are in the .gitignore file. I just commited one, it's in the project root. You can refresh your branch with the master.
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: Evolution of the SD card API

Post by CunningFellow »

The mmc.s was compatible with the original mmc.s but I removed two functions just for this post

That is

init (which was a wrapper around init_no_buffer)

and the function to read a whole 512 byte sector

The only incompatibility or issue with intermixing this and petiteFS/FatFS is that you must call stop_transmission before trying to do non-streamed reads.

I'll see if I can add the seek, read_to_memory_with_spanning and the non blocking getChar before I next scratch my head at git.
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: Evolution of the SD card API

Post by D3thAdd3r »

That small flash footprint is amazing. That will make a huge difference combined with the wide variety of supporting functions you mention.
Post Reply