Condusiv Technologies Blog

Condusiv Technologies Blog

Blogging @Condusiv

The Condusiv blog shares insight into the issues surrounding system and application performance—and how I/O optimization software is breaking new ground in solving those issues.

Optimizing Cameras, GPS, Smart Phones and other SD Card devices

by Michael 9. January 2010 09:36

Rumack: Fragmentation slows down SD Card performance!

 

Striker: Surely you can't be serious.


 

Rumack: I am serious... and don't call me Shirley.    

 

The word is slowly [but surely :-)] getting out that fragmentation affects NAND Flash storage - specifically free space fragmentation. The SD Association website is a great resource for edification and development resources. On their page describing the SD Card Speed Classes, they even discuss fragmentation.

 

 

Fragmentation and Speed

The memory of a card is divided into minimum memory units. The host writes data onto memory units where no data is already stored. As available memory becomes divided into smaller units through normal use, this leads to an increase in non-linear, or fragmented storage. The amount of fragmentation can reduce write speeds so higher SD card speeds help compensate for fragmentation.

 

There are several methods to address this issue. One is, as mentioned, buy better performing storage and hope your requirements never exceed the cards ability to deliver on your needs (though fragmentation will still limit the storage's peak performance). However, a better approach is to fix the root issue, and there are two ways to do that.

1. Defragment the free space (e.g. HyperFast, Diskeeper)

2. Copy all the data off the SD Card and reformat the card

The best approach is likely to be determined by the device in which the SD Card is used. If it's from your digital camera, option 2 is probably pretty easy to undertake. If pulling all the data off the card is not feasible, optimize it.

The only other question then might be "how often do I run optimization?". I did a blog post on that, including some performance tests, a few years ago, Read that Here or Here.

Lastly, I thought I'd include a recent personal success from an IT professional who happens to also use Diskeeper at work:

"Happy new year!! I made an interesting test over the holidays using Diskeeper 2010. I was updating my GPS with new maps and discovered that my Garmin unit uses Fat32 file system so I figured I would run your software on it just for fun and see the results. I was able to defragment a large part of the files and it more than doubled the speed of the unit. The images render faster with less stutter, routes are recalculated almost instantly now and it finds points of interest much faster. I did the same test on 2 other GPS`s and got the same results." Regards, Carlo

Tags:

Defrag | SD Card | SSD, Solid State, Flash

The Impact of Fragmentation on Flash Drives (iPods, Jump Drives, etc)

by Michael 22. June 2007 12:50

One of the questions that comes up on occasion is "should I defrag my iPod, my SD card, or my USB drives?". To answer that, let's first take a step back and make note that these drives (also known by other names such as thumb drives, jump drives, solid state disks, etc) are flash-based storage devices (the largest I've seen is 32GB). They are used in Digital cameras under the names of SD cards, compact flash, memory stick, etc. The iPod and other MP3 players either have miniature hard disk drives (HDD), typically called microdrives, in the larger storage models, or flash-based drives in the smaller, 2Gb-4GB, such as with the iPod Nano. The exact nomenclature of a flash storage device depends on it's "interface". If it uses a USB interface it is typically called a jump drive, if it uses a SATA/SCSI interface and is intended to replace a hard disk drive, it is called a Solid State Drive (SSD). Other flash devices include the aforementioned digital media storage devices such as Memory Stick, Compact Flash, etc... In a nutshell, Flash based disks do not use a spinning disk and can access data randomly without any performance penalty. That may seem to obfuscate the benefit of defragmentation, and to a good degree it certainly mitigates the need. Flash and SSD devices are good at reading data, but are not as good at writing data. The reason for the poor write performance is that these (NAND based) devices must erase the space used for new file writes, immediately prior to writing the new data. This is known as erase-on-write or erase/write. Improvements in this area are coming (phase-change memory). However, flash devices running FAT or NTFS file system do still fragment the same way that a HDD would. Non-Windows products, like digital cameras/camcorders, use the FAT file system (FAT16 or FAT32, depending on the size of the drive). FAT file systems are more susceptible to fragmentation than NTFS. The greatest drawback of flash devices from the perspective of fragmentation is it is slow at random write I/O. Here's a quick test you can do yourself to show that severe free space fragmentation on Flash drives does affect performance. I did this myself, as a test run.

I took a brand new Kingston 1GB DataTraveler Hi-Speed USB drive with 24MB/sec read and 10MB/sec performance (per the manufacturer).

First I did a format of the disk - FAT16 (you'll need to use FAT32 for drives over 2GB). Then, using a development testing tool from Diskeeper Corp I fragmented the free space. I used Diskeeper 2007 to confirm the fragmentation as well as DiskView (a more granular tool available from Microsoft - formerly SysInternals). I created about 45MB of small files (16k to 48k in size) spread all across that Flash disk.

I then grabbed the VM Player install file (145MB), and made five more copies of it and zipped (Winzip) them into single 846MB zip file. This file was kept on a separate spindle (SATA disk) from the OS and paging files (to minimize variables from my time tests).

I used a simple stop-watch to time how long it took to copy this file from the SATA disk to the USB Flash drive with fragmented free space. It took 2:37 from start to finish.

I reformatted the USB drive, to the FAT16 file system again and rebooted the PC (just to make sure the cache was clear). I then copied that 846MB zip file from the same location over to that USB drive. This time the copy operation took 1:14, less than half the time required to copy than when the free space was fragmented.

Deleting a large, fragmented file also takes a long time.

From a "scientific" perspective the test can be run a few more times to come to an average, but given the difference was so significant, I personally did not feel the need to redo it. You can reverse the test order, and even use a program to zero-out the flash drive, just to eliminate any minor possible variables. Anyone else is certainly welcome to give this a go for themselves.

I did test one more case where I fragmented the free space into 24 even chunks and found no difference in copy time. While severe free space fragmentation is an issue, mild free space fragmentation is not - same concept as on physical disk. And yes the 846MB was fragmented in 19 pieces.

To create the free space fragmentation (without the development tool I'm privileged to have access to), you can copy a large number of small files to the Flash drive and deleted every other one, or other random deletion pattern (vary between deleting every third, fourth, fifth ...n file). If you have some programming skill this can be
scripted fairly simply. Just make sure there is enough room left on that USB drive after fragmenting the free space to copy the same test file. The more severe the free space fragmentation, the longer the copy operation will take.

That said, the degree to how this translates into actual usage depends. A real-world equivalent might be with a digital camera/camacorder where you mix various sized mpegs and jpegs, and use the device to delete some of these files from the drive. Unless you wipe the disk, the free space fragmentation will build up.

The test case I made up may be so extreme that it is unreal. I don't know what's "real-world" as I don't personally use Flash drives that often, and even then my actual usage isn't likely to equal yours. How often you want to consider free space consolidation depends; my best-guess is once every 6 months or so. The limited extent to which I use USB drives and the fact my 2GB mp3 player only ever gets minor and infrequent file changes, I doubt I'll personally ever need to worry about the free space fragmentation.

PS: We've been working with several of the technology leaders in the Flash/SSD industry for some time. They have been kind enough to send us pre-release devices for our R&D efforts. Expect future innovations from Diskeeper Corporation and those industry partners to improve performance and reliability on these storage devices.

Tags:

General | SSD, Solid State, Flash

Month List

Calendar

<<  April 2018  >>
MoTuWeThFrSaSu
2627282930311
2345678
9101112131415
16171819202122
23242526272829
30123456

View posts in large calendar