SD Card Defrag

Topics related to the API, programming discussions & questions, coding tips, bugs, etc. should go here.
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: SD Card Defrag

Post by Artcfox »

I don't know that cameras know how to format it optimally either. I think that tool might be the only optimal way, unless to be certified, the people making the camera have to agree to implement formatting according to their spec (under NDA).
User avatar
danboid
Posts: 1937
Joined: Sun Jun 14, 2020 12:14 am

Re: SD Card Defrag

Post by danboid »

I'm sorry if you've already posted it elsewhere on the forum Artcfox but have you done any testing under Linux or on a PC of somesort to verify their claims? Does the sdcard.org formatting tool really make a noticeable difference for more mainstream use cases or have you only done Uzebox related tests?

I'm trying to decide how best to reply to the dosfstools maintainer who wants me to be more specific about my request for improved SD card support in mkfs.vfat etc. I haven't tried the sdcard.org formatting tool myself yet, I only ever use Windows as a last resort :D

Have you got any suggestions for how they might implement this feature?
CunningFellow
Posts: 1445
Joined: Mon Feb 11, 2013 8:08 am
Location: Brisbane, Australia

Re: SD Card Defrag

Post by CunningFellow »

I can confirm that the SD card tool formats in a way that makes cards work faster than the format tool built into at least Windows 7.

Newer versions of windows may have improved that but Win7 does it sub optimally.

I have not tried recent cameras but I also know that an old Casio I had formatted SD (under 2GB) optimally and the camera actually stated in the user manual to use the build informat rather than an OS one.
User avatar
danboid
Posts: 1937
Joined: Sun Jun 14, 2020 12:14 am

Re: SD Card Defrag

Post by danboid »

How long do you think it'd take you to RE this CF?
User avatar
Artcfox
Posts: 1382
Joined: Thu Jun 04, 2015 5:35 pm
Contact:

Re: SD Card Defrag

Post by Artcfox »

There is now an official sdcard.org formatter for Linux: https://www.sdcard.org/downloads/sd-mem ... for-linux/

I haven't tried it yet, but this makes me happy.

Edit: Interesting discussion on Hacker News about it and what it might be doing under the hood: https://news.ycombinator.com/item?id=35610243
User avatar
danboid
Posts: 1937
Joined: Sun Jun 14, 2020 12:14 am

Re: SD Card Defrag

Post by danboid »

Thanks Artcfox!

The highlights from the HN comments:

dmitrygr 2 hours ago | prev | next [–]

So much confusion in this discussion. Let me try to clarify (all of this is very very simplified).

1. NAND in SD cards has page size (minimum writeable piece) something like 4kB or 8KB, or maybe even 64KB. But as per spec SD cards must allow writes at 512 byte granularity. Aligning your file system clusters with these flash pages will allow SIGNIFICANTLY better performance and lower wear. How? Well in the CSD register (accessible using SD commands), the NAND geometry is given, so all it takes is reading that and creating a file system that is properly aligned.

2. Why not just mkfs.vfat? You can, but unless you do some of the above work yourself and some math, you’re unlikely to get the alignment right and will suffer because of it, especially on small or random writes. You CAN do it yourself. Just takes a little work.

3. What about my gopro, my phone, or $DEVICE-X that can format SD cards? Well, most device firmwares DO indeed do this correctly as the SD spec advises to do.

4. Why doesn’t the card just do it in firmware all by itself? Curiously sony memory stick does this. Why not sd? There isn’t a command in the spec for that and adding it now is too late - nobody will use it. Thus we have this tool and ones like it.

5. What is this about the secure area? The tool just says it doesn’t touch it. Almost nobody does. Costs a lot of money to license the SD security stuff. Nobody uses it. It was meant for securing MP3s on cards. Secure area exists on all cards (yes you are paying for flash you cannot use). Luckily it isn’t big. Special commands are used to access it. This tool and basically any device you’ll ever encounter in your everyday life do not use them. Lacking those commands, it is as if the secure area doesn’t exist.

6. Why dont windows/Linux/macOS do it right themselves? My guess? Separation of concerns. The formatting code there only formats a partition. For proper alignment you may need to start the partition on the right block as well. Simple example. Say in your fictional card, we have 4KB pages. That’s 8 512-byte sectors. So we want our first FAT cluster to start at a multiple of 8 (and be a multiple of sectors in size). Say we work out that our various FS Structures need 17 sectors. That means that we need the partition to start at sector 7. Thus we make the MBR do so. Wait! You might say that FAT already has “resevered sectors” value we can use for this. No MBR touching needed. Well, you’d be surprised how many devices/firmwares break with nonstandard values there. Also, what makes you so sure that mkfs.vfat can even find out how many sectors into /dev/sdb, /dev/sdb1 starts, to do the math right? For that matter, what makes you think it would even know to check for SD-card-ness of /dev/sdb?
Someone also linked to this guide on how to properly partition and format microSD cards under Linux using open source tools:

https://web.archive.org/web/20230405090 ... -lifetime/
User avatar
danboid
Posts: 1937
Joined: Sun Jun 14, 2020 12:14 am

Re: SD Card Defrag

Post by danboid »

I have updated my dosfstools feature request with what we've learned from the HN comments section.

https://github.com/dosfstools/dosfstools/issues/193
User avatar
D3thAdd3r
Posts: 3222
Joined: Wed Apr 29, 2009 10:00 am
Location: Minneapolis, United States

Re: SD Card Defrag

Post by D3thAdd3r »

Great news all around :ugeek:
Post Reply