SD Card Defrag

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

SD Card Defrag

Post by D3thAdd3r »

I've off and on thought about the need for high performance SD stuff to have a reasonable guarantee on read speeds. The bootloader API is awesome in general, but there are legitimate situations where having to follow the chain to find the next sector in a file(I think bootloader 5 does this?) can't be done. If the card is not fragmented, this is no issue.

Defragmenting a card seems to be achieved by deleting files and putting them back on, as far as I know. However easy this is, I don't know if it's necessarily intuitive to a new user. They would just see a game fail, assume it's broken, and not play it. Hopefully they would find the forum/wiki posts, who knows. If the games could detect fragmentation and point the user to a tool(or better, optionally bootloader reset to it) this would be ideal.

So my questions:

*is there a reasonably low resource and fast method for a game to detect if it's data file is fragmented?

*would the amount of time necessary to defragment a card even be realistic at Uzebox speeds?

*could this only be done with SPI RAM(or would that not be enough)?

*in general how would this be done? There must be a lot of sector swapping to do it?

Uzenet might offload the whole image to a server(potentially hours at 115200...), erase the card, and receive/write the unfragmented version. This seems very slow and likely to drop out. Any other ideas or it the entire concept not feasible at all?
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: SD Card Defrag

Post by Artcfox »

In order to get the max read speeds, you need to have the starting sector of the FAT partition aligned with the internal hardware implementation. The methods for determining that are proprietary and protected with a license (probably to ensure there is one and only one implementation that gets updated whenever the SD spec is revised). That tool is at sdcard.org and you need to agree to that license to download it. When I have run it on newer cards it sometimes adds 16MB of padding before the FAT filesystem to align it with the internal hardware on the SD card, but other cards from the same manufacturer get different alignments with the tool. So it is doing some black magic for sure to optimize things, but it matches the format the card had from the factory. A format from Windows or Linux will not necessarily match the optimal format and read performance may suffer by a ton. The difference is enough to make reading slower by orders of magnitude, so it really is worth it to use that tool.

It formats the card, so using it will get you a blank SD card so you can just copy things off before and then back on after using it, and it should be the max speed possible with no fragmentation. I don't think there is a better way.
User avatar
danboid
Posts: 1931
Joined: Sun Jun 14, 2020 12:14 am

Re: SD Card Defrag

Post by danboid »

I'd never heard about this official sd card formatting tool until you mentioned it recently artcfox. Its annoying there is no Linux version of it and if it makes as big a difference as you state then you have to wonder why MS and Apple haven't licensed and integrated it into Windows and macOS. I've never seen any guides anywhere recommend formatting with this tool but I suppose in most cases for most (non Uzebox) users a bit of fragmentation doesn't matter.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: SD Card Defrag

Post by Artcfox »

danboid wrote: Sun Mar 12, 2023 6:26 pm I'd never heard about this official sd card formatting tool until you mentioned it recently artcfox. Its annoying there is no Linux version of it and if it makes as big a difference as you state then you have to wonder why MS and Apple haven't licensed and integrated it into Windows and macOS. I've never seen any guides anywhere recommend formatting with this tool but I suppose in most cases for most (non Uzebox) users a bit of fragmentation doesn't matter.
It is not so much about fragmentation as it is about alignment. The cards come from the factory aligned. Only if you reformat them with another tool would the alignment change. Without licensing anything, a smart tool (on any OS) could look at the partition locations before formatting and ensure any new partitions are created with that same alignment, but I don't know of any tool that does this.

You can download and read the SD (simplified) specification on sdcard.org. For any of the faster (non-SPI) modes you need to sign an agreement in order to download the specification.
User avatar
D3thAdd3r
Posts: 3221
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: SD Card Defrag

Post by D3thAdd3r »

Good info, thank you. I remembered that tool existed, but didn't know it did optimizations even the OS can't necessarily match.

Seems the only reasonable way to maximize performance is with the official tool. This is good, since I can put this idea/question out of my mind permanently!
User avatar
danboid
Posts: 1931
Joined: Sun Jun 14, 2020 12:14 am

Re: SD Card Defrag

Post by danboid »

The king of the feature request returns:

https://github.com/dosfstools/dosfstools/issues/193
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: SD Card Defrag

Post by Artcfox »

danboid wrote: Tue Mar 14, 2023 9:36 am The king of the feature request returns:

https://github.com/dosfstools/dosfstools/issues/193
Nice! I wonder if it would be possible to collect data on a bunch of cards and get some insights, or just always align to the largest alignment we see on a reputable (non-counterfeit) OEM (SanDisk or Samsung) card. All the others manufacturers I believe buy the flash chips rather than produce them in house.
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: SD Card Defrag

Post by CunningFellow »

Doesn't the card tell you itself the memory layout via the extended CID request thingy?
User avatar
danboid
Posts: 1931
Joined: Sun Jun 14, 2020 12:14 am

Re: SD Card Defrag

Post by danboid »

CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: SD Card Defrag

Post by CunningFellow »

Sorry - It is in the CSD not the CID regsiter. Field called erase sector size.

Though in my quick google search to find my mistake I did see something that said this is only valid for SD card and the XC and HC cards use a different approach.

The fact that cameras and other devices know how to format the cards for optimum speed means that the info isn't that guarded a secret.

Windows built in format tool obviously just does not care.
Post Reply