Condusiv Technologies Blog

Condusiv Technologies Blog

Blogging @Condusiv

The Condusiv blog shares insight into the issues surrounding system and application performance—and how I/O optimization software is breaking new ground in solving those issues.

Best Practices for Storage Area Network (SAN) Defragmentation

by Michael 29. March 2011 02:30


As high performing storage solutions based on block protocols (e.g. iSCSI, FC), SANs excel at optimizing block access. SANs work at a storage layer underneath the operating systems file system; usually NTFS when discussing Microsoft Windows®. That dictates that a SAN is unaware of “file” fragmentation and unable to solve this issue.

Fig 1.0: Diagram of Disk I/O as it travels from Operating System to SAN LUN.

With file fragmentation causing the host operating system to generate additional unnecessary disk I/Os (more overhead on CPU and RAM) performance suffers. In most cases the randomness of I/O requests, due to fragmentation and concurrent data requests, the blocks that make up the file will be physically scattered in uneven stripes across a SAN LUN/aggregate. This causes even greater degradation in performance.

Fig 1.1: Sample Windows Performance Monitor Report from fragmented SAN-attached NTFS volume.

Fortunately there are simple solutions to NTFS file system fragmentation; fragmentation prevention and defragmentation. Both approaches solve file fragmentation at the source, the local disk file system.

IntelliWrite® “The only way to prevent fragmentation before it happens™”

IntelliWrite is an advanced file system driver that leverages and improves upon modern Windows’ file system “Best Fit” file write design in order to write a file in a non-fragmented state on the initial write. Intelligently writing contiguous files to the disk provides four principal benefits above and beyond defragmentation, including:

  • Prevents most fragmentation before it happens
  • Better file write performance
  • An energy friendly approach to improving performance, as defragmentation is not required for files handled by IntelliWrite
  • 100% compatibility with copy-on-write technologies used in advanced storage management solutions (e.g. snapshots)

While eliminating fragmentation improves performance. it is important to properly configure and account for advanced SAN features.

With the increasing popularity of SANs, we've included instructions in the Diskeeper installation to ensure users properly configure Diskeeper:

We suggest reading this full document before executing any of the recommended configurations. These instructions apply to V-locity (used on VMs as well).

Best Practices:


Implementing Diskeeper on a SAN is simple and straightforward. There are two principal concepts to ensuring proper configuration and optimal results:

  • Ensure IntelliWrite is enabled for all volumes.
  • Find a time to schedule Automatic Defragmentation (more details below)

If you are implementing any of the following SAN based technologies such as Thin Provisioning, Replication, Snapshots, Continuous Data Protection (CDP) or Deduplication, it is recommended to follow these guidelines.

Defragmentation can cause unwanted side effects when any of the above referenced technologies are employed. These side effects include:

With SAN replication:
Likelihood of additional data replication traffic.

With Snapshots/CDP:
Likelihood of additional storage requirements for data that defragmented/moved and snapshot-related performance lag.

With Thin Provisioning:
Likelihood of additional storage requirements for data that defragmented/moved.

With Deduplication:
Potential for additional deduplication overhead. Also note that deduplication can be used to remove duplicate blocks incorrectly allocated due to defragmentation. This process can therefore be used to reclaim over-provisioned space.

This is why it is important to enable the fragmentation prevention (IntelliWrite) and change the Automatic Defragmentation to occur during non-production periods to address the pre-existing fragmentation:

During Installation, disable Automatic Defragmentation;

Uncheck the “Enable Automatic Defragmentation” option during installation.

Upon installation ensure IntelliWrite is enabled on all volumes (default). IntelliWrite was specifically designed to be 100% compatible with all advanced SAN features, and should be enabled on all SAN LUNs. IntelliWrite configuration is enabled or disabled per volume, and can be used in conjunction with Automatic Defragmentation, or exclusively.

To ensure IntelliWrite is enabled, right click a volume(s) and select the feature.

Then confirm “Prevent Fragmentation on this volume” is selected, and click “OK” to complete.

Once installed, enable Automatic Defragmentation for any volumes that are not mapped to a SAN LUN. This may include the System Partition (e.g. C:\).

To enable Automatic Defragmentation, right click a volume(s) and select the feature.

Then check “Enable Automatic Defragmentation on the selected volumes” and click “OK” to complete.

If you are not using any advanced SAN features, it is recommended to enable Automatic Defragmentation for all days/times. However, note that pre-existing fragmentation will require significant effort from Diskeeper to clean up. This effort will generate disk I/O activity within the SAN.

Therefore, if existing fragmentation is significant, initially schedule Diskeeper to run during off-peak hours. As Diskeeper has robust scheduling capability, this is easily configured.

To enable Automatic Defragmentation during non-production periods, right click a volume(s) and select the feature.

Then check “Enable Automatic Defragmentation on the selected volumes”. Diskeeper is then scheduled by using your mouse to highlight over the 30 minute blocks in the interactive weekly calendar.

The above example disables defragmentation Monday through Friday. It also disables defragmentation Saturdays and Sundays except between 7pm until 3:30am the following morning. This would afford 17 hours of defragmentation availability per week. Immediately following these scheduled defragmentation periods is when SAN maintenance for advanced features should be addressed (e.g. thin reclamation, deduplication).

Should accommodating SAN maintenance be difficult (e.g. limited maintenance windows)using a weekly optimization process, very granular scheduling is also available with Diskeeper. Note, maintenance windows are not required in order to implement and benefit from IntelliWrite.

To schedule for specific non-reoccurring dates and times in the future, select the “Turn Automatic Defragmentation on or off based on specific dates” option. Click any multitude of dates and times using Shift-Select or Ctrl-Select. Once done, click OK to complete.

If you are implementing the above mentioned advanced technologies and your SAN provides hot block optimization / data tiering, it is also recommended to disable I-FAAST® (Intelligent File Access Acceleration Sequencing technology). I-FAAST sequences hot “files” (not blocks) in a Windows volume, after determining hardware performance characteristics. The sequencing process creates additional movement of data for those advanced SAN features, and is therefore generally recommended to disable when similar SAN solutions are in place.

To disable I-FAAST, right click a volume(s) and select the feature.

Note, I-FAAST requires Automatic Defragmentation be enabled. Also note that I-FAAST is disabled by default in Diskeeper 2011 in certain cases. Also note that I-FAAST generates additional disk I/Os and will therefore cause an increase in the aforementioned Automatic Defragmentation side effects.

Once pre-existing fragmentation has been removed, increase the periods in which Diskeeper actively optimizes the Windows file systems. With real-time defragmentation and InvisiTasking® technology, Diskeeper immediately cleans up fragmentation (that is not prevented by IntelliWrite). This minimal ongoing optimization generates only invisible, negligible I/O activity.

New features in Diskeeper 2011 to improve SAN performance:

Diskeeper 2011 introduces SAN specific solutions. These default solutions automate many of the configurations required for SAN-attached servers.

Diskeeper 2011’s new Instant Defrag™ technology dramatically minimizes I/O activity, and exponentially speeds up defragmentation. The Instant Defrag engine is provided fragmentation information, in real-time, by the IntelliWrite file system filter driver (those fragments that it does not prevent). Without the traditional need to run a time and resource intensive whole-volume fragmentation analysis, Instant Defrag can address the recently fragmented files as they occur. This dynamic approach prevents a buildup of fragmentation, which could incur additional I/O overhead to solve at a later date/time.

Diskeeper 2011’s new Efficiency Mode (default) maximizes performance, while minimizing disk I/O activity. By focusing on efficiency and performance and not on presenting a “pretty disk” visual display, Diskeeper 2011 minimizes negative side effects (e.g. reduce snapshot storage requirements or thin LUN growth, etc..) while maximizing performance benefits. It is a SAN-optimized defrag mode and our recommended solution for SAN-attached Windows volumes.

By default, Efficiency Mode also disables proprietary file placement features such as I-FAAST.

Also, by default, Diskeeper 2010/2011 moves data to lower NTFS clusters, and hence generally “forward” on SAN LUNs.

Best Practices Summary:
  • Ensure IntelliWrite is enabled for all volumes.
  • Automatic Defragmentation should be enabled at all times for all direct attached storage volumes.
  • Use Efficiency Mode of Diskeeper 2011.
  • Schedule Automatic Defragmentation on SAN LUNs, based on use of advanced SAN features.
  • Run SAN processes such as space reclamation and/or deduplication on recently defragmented LUNs using advanced SAN features.

Want this in PDF form. Get it here: Best Practices for using Diskeeper on Storage Area Networks.pdf (3.00 mb)

Tags: , , , , ,

Defrag | Diskeeper | SAN

Do you need to defragment your SAN?

by Michael 11. January 2011 13:04

I recently came across an older article about defragmenting SANs (read it here). It includes interviews with analysts, SAN vendors (some pro-defrag, some against), and an employee from Diskeeper Corporation.

I was particulary impressed with the EMC'ers response:

"The SAN can't do anything about the fact that Windows sees the file in 30 bits," said Wambach. "That's really something that is happening outside of the storage realm."

He highlights the abstraction perfectly.  SAN vendors claim that a defragmenter cannot correct fragmentation due to the fact it is abstracted from the physical blocks. We absolutely agree with this statement. And for that same reason, SANs cannot fix fragmentation in the NTFS file system, which causes excess and unnecessary overhead on the OS.


Inside SSDs 101

by Michael 31. December 2010 06:08

We have numerous partners and alliances in the  Solid State drive (SSD) space that we interact with regularly. Conversations in related meetings with those allies continues to revolve around the same issue, overcoming performance bottle necks at the storage level. In attacking this problem to get higher performance for things like boot times, application load times, etc., the industry has turned to flash memory, otherwise referred to SSDs (we'll also be announcing a brand new SSD technology in the next few weeks).

The following may be well known to those highly knowlegdable in SSDs, but hopefully helps others less versed in their design.

High end SSDs have proven to yield some very impressive read times, well over double a typical SATA hard disk drive in some cases.

Here are some example published speeds from a few manufacturers/models:

Seagate 7200.12 HDD 500GB, 750GB, and 1TB family

Read/Write speeds (outer tracks, empty drive): 125MB/sec sustained, 160MB/sec peak

Intel X25-M 80GB SSD (MLC)

Sequential Access - Read: up to 250MB/s

Sequential Access - Write: up to 70MB/s

Intel X25-M 120GB SSD (MLC)

Sequential Access - Read: up to 250MB/s

Sequential Access - Write: up to 100MB/s

Intel X25-E 32GB SSD (SLC)

Sequential Access - Read: up to 250 MB/s

Sequential Access - Write: up to 170 MB/s

One of the main reasons for such fast read times is the lack of “seek time” that an SSD has to find and retrieve a piece of data versus a hard drive.  Simply put, a hard drive has to move a magnetic head connected to an arm over a track on a platter and then through various means find the data requested and read or write something.

Now you have to admit, a hard drive does this quite well and very fast, considering the physics involved.

On the other hand an SSD sends an electrical pulse to read the data which is much faster in comparison; give or take on the order of magnitude, double on higher end SSDs. The lack of a moving part cuts the time down considerably.  

Now, writing data to SSDs is a whole other story which leads us down a bit of a rabbit hole so to speak and the main subject of this blog.


To start with, let’s look at what an SSD is:

Note, I have borrowed some photos and descriptions from the following site:

Firstly you simply have a small piece of silicon with a whole lot of transistors that look like the following:


Each transistor is 1000 times thinner than a human hair. In essence, each transistor either holds an electrical charge or it doesn’t.  In the case of SSDs, a transistor that is charged equals the value 0 and the ones that are not equal a value of 1. That is the extent of it. 

In the above photo the transistor that is charged has a circle around the “e” which stands for electrons representing the charge.   

Now, to read what’s inside these transistors an electrical pulse is sent to them and by reading the signal from the pulse sent it’s able to tell which are charged and which are not. All in all this is a pretty fast operation. Even writing to a transistor for the first time is pretty fast as it’s empty with no charge to begin with. 

BUT…… what happens when you have to write to an area of the SSD that has already been written to?  Here in lays the problem and the unfortunate “Achilles heel” for SSDs. With a hard drive you can just simply flip the bit in place and call it a day.  In fact this is a faster operation on a hard drive then it is on an SSD, over the long run. Now aside from the extra step it takes to erase an SSD it gets a lot worse. But in order to understand this we need to view how data is laid out on an SSD:

The smallest unit is a single transistor, also known as a cell. Imagine the dot shown is a top view of a single transistor magnified a gazillion times.

This single transister holds a single bit of data, i.e. a 1 or a 0.  The next unit size up is a called a page and holds 4KB of data.   To put this in perspective, there are 8,192 bits in one kilobyte so each page contains 32,768 transistors.   A page is the smallest unit that can be written to on the SSD.  This means that even if the data you are writing is only 1,500 bits in size, it will use up the entire 4KB of space and make it unusable for writing other data.  You only get to write once to it until its been erased for reuse again.

In fact, to update the data within this page, the data would have to be copied first, updated and rewritten to a new page leaving the old page unusable till it’s been erased.  The controller has to earmark it for clean up so it can be used again later.

Now, even though this page has been earmarked for being erased, it may not get erased for some time because it’s actually part of a bigger group of data called a “block”.  A block is a group of pages as illustrated below:



The number of pages that make up a block may vary from one SSD model to another and can be very large (in the megabytes).  On one such SSD we have tested on, this happens to be 128k, which would be a group 32 pages.  This data block (32 pages) is actually what’s called an “erase block”.  An SSD only erases one data “block” at a time.  So back to our example of this page with old data in it: theoretically this page could sit around a while before this block of data is erased by the SSD.  It’s plausible this could have in some cases security of data issues but that’s a subject for research and test. 

Now, when data is updated in a page as discussed earlier it has to be moved to another location unbeknownst to the file system and internally mapped by the controller on the device to keep track of it as illustrated below:

There is obviously overhead associated with this. So a page, for argument sake, could have three states of existence:

Ready to be used (it’s erased or been erased thus all transistors have been set to 1) as in the clear little boxes above;

Used -- as in the blue boxes above;

Dirty (containing invalid data that needs to be erased) as in the black boxes above.  

The overhead in handling dirty data is huge, and referred to as garbage collection.  Garbage collection is the process of moving good data out of areas where old data exists in order to erase the old data in order to reclaim pages that can be written to again.

Doing a lot of this garbage collection activity creates a lot of activity called “write amplification”.

This is the disease that SSDs are plagued with which kills their write performance, particularly as the device fills up.  It’s also what shortens the life span of the device.

The following is a definition or description from Wikipedia that I think is pretty good:

“Write amplification (WA) is a phenomenon associated with Flash memory and solid-state drives (SSDs). Because Flash memory must be erased before it can be rewritten, the process to perform these operations results in moving (or rewriting) user data and metadata more than once. This multiplying effect increases the number of writes required over the life of the SSD which shortens the time it can reliably operate. The increased writes also consume bandwidth to the Flash memory which mainly reduces random write performance to the SSD. Many factors will affect the write amplification of an SSD, some can be controlled by the user and some are a direct result of the data written to and usage of the SSD.”

Now, there is a comment that is made in the above description that is interesting, and could lead to read performance in some cases getting degraded, which is the “increased writes” consuming bandwidth to the Flash memory interrupting a read operation.

Now, I don’t say this as a fact but rather postulating whether or not reads are affected. 

The overhead in writes required by the Windows NTFS file system to do just one IO could be considered extreme from the SSD’s point of view.  To create and write one bit of information, such as a notepad doc with a single number in it, requires an update to the MFT file by creating a record of the operation, update to a directory file and any other metadata files such as journal files that are keeping track of operations at the time. The point is, for every one write of user data there are several writes occurring to keep track of it. 

Current file systems were designed based on hard disk drive characteristics, not SSDs.  From the SSD’s point a view, NTFS writes with wild abandon. This puts a lot of overhead on the SSD controllers with a lot of overhead on data mapping and housekeeping at the controller level of the SSD which hard drives don’t have to worry about much.  A hard drive typically only has to re-map data when it has a bad sector on it as an example.

NTFS file system thinks for example that a file should be split in two (split IOs) because it thinks it doesn’t have a contiguous free space for the file that is being updated.  Yet the SSD may have, during its garbage collection process created a space or remapped data clusters but the operating system doesn’t know this and vice versa. 

The current TRIM functionality is supposed to help SSDs within Windows 7 but it's far from being a panacea to the write amplification issue. 

Different types of SSD

SLC stands for “single level cell”

MLC stands for “multi level cell”

TLC stands for “tender loving care". okay not really, just checking to make sure you are paying attention. It really stands for" tri level cell”; really. 

SLC is faster than MLC and TLC. 

The design of an MLC and an SLC are pretty much the same.  The difference is an MLC is able to put more than one value in a single transistor (referred to as a cell) by layering the data within the cell.  Typically two or more bits are able to be placed in a single cell with MLC versus one bit in SLC. 

So MLC is able to contain twice as much data versus SLC.  That’s the plus side.  The down side is MLC in reading and writing data to a single cell has to be very precise and has a lot of overhead logic wise, so MLC ends up slower than SLC due to the preciseness that is required to determine if two values exist and what those values are in a single cell. Also the life cycle of MLC becomes 10x shorter than SLC.  The following is a great white paper that describes the differences very well and how voltages are used to read the values, etc.

The difference between TLC and MLC is NOT more transistors.  L stands for "level", referring to voltage level to a transistor, not multiple levels or numbers of transistors.  Again the above link along with a few other sites lays this out fairly well. 

The difference between one flash memory or SSD is not so much the chip itself but the supporting controller and its design.  This includes creating multiple data channels, on-board Raid and other fancy and expensive solutions.  The other differentiator is software within the controller managing all of the mapping of data, moving data around, etc.  Sounds like the age-old fragmentation problem to me again; just at a slightly different level.

Thin Provisioning and Defrag

by Michael 30. November 2010 08:55

Before I cover considerations and recommended configurations in thin provisioned storage environments it’s important to revisit why defragmentation of Windows operating systems is so important in a virtualized machine and/or virtualized storage environment.  

The problem is that fragmented data in a local disk file system, such as NTFS, causes the operating system to generate additional I/O requests. For each “logical” fragment in the file system, a separate I/O request packet (IRP) must be generated and passed on to underlying storage layers. So for example, a file in 100 fragments would generate 100 separate smaller I/Os, rather than a single larger I/O.  

This translates to an operating system processing a great deal more unnecessary I/O traffic, thereby increasing CPU and memory demand. In many cases that excess I/O is passed on to a Storage Area Network (SAN) and/or virtualization platform, causing additional unnecessary overhead.

In some cases, data that is in a contiguous sequence of clusters in a local disk file system will be physically contiguous on the actual storage media, i.e. the disk drive/array. This is generally a valuable added benefit, but by no means required for defragmentation to greatly increase performance.  

Some file systems (e.g. log-structured file system) used in SANs may intentionally fragment data at the “block” level. They may coalesce random writes from the OS into sequential writes within the storage. While this will minimize I/O activity in the SAN, it actually increases the likelihood that the data in those sequentially written stripes is physically fragmented, because the coalescing process is not based on re-ordering of blocks as they map to a common file – it simply dumps the data to the media. For these environments, you’ll need to check with your storage vendor regarding proprietary defragmentation solutions for their SAN.  

Regardless of spatial proximity, the benefit of a fragment-free local disk file system (NTFS) is that your OS and virtualization platforms aren’t processing extra I/Os generated, due to fragmentation, and will therefore be able to host more operating systems and process more data, faster.   

Thin Provisioning 101 

Thin provisioning allocates resources from an aggregate storage pool which is essentially divided into assignable units commonly referred to as ‘chunks’. Provisioning storage in ‘thin’ environments is done in chunks that are pulled from that pool of available, and as yet unallocated, storage.

As data is added to a thin provisioned container, such as a Dynamic/Thin virtual disk or a LUN, that container increments, usually in a just-in-time basis, by a chunk or number of those chunks, depending on how many chunks are needed to house all the incoming writes. A chunk can be anywhere from a few kilobytes to gigabytes in size, and varies from one thin provisioning technology vendor to the next. In some cases it is a fixed size, in other solutions the chunk size is user-selectable. 

How and when chunks are allocated also varies from vendor to vendor.  

Many thin provisioning technologies provision for every write. They monitor blocks, and specifically changes to blocks. As new data is written, space is provisioned for it on a just-in-time basis, and it is stored.  


Another method to provision space is based on the Windows volume high water mark. A high water mark, with respect to a volume in this definition, is the term that describes the last written cluster/block of data (the highest used Logical Cluster Number, or LCN, on the volume). Everything beyond the high water mark is assumed to be null.   

NTFS Write and Delete Design 

While not exactly “thin friendly”, NTFS is undeserved of the reputation of being a problem for thin provisioned disks/LUNs. It has been mistakenly stated that NTFS carelessly writes to continuingly new and higher LCNs, until it has written to every cluster on the volume, before circling back around to clusters since freed up from file deletes. This is not correct. 

When describing NTFS design as it relates to storage provisioning, we should first describe the various file sizes. There are three sizes for files in NTFS, and they use high watermarks too. 

The Valid Data Length (VDL) is the distance into the file that data has actually been written, as it resides in the cache. It is depicted as the blue bar in the diagram. A VDL can include sparse runs interspersed between data. The highest written LCN that constitutes the VDL is the high watermark for that file. There is no data, at least related to this file that resides past the high watermark. Without having to actually write zeroes, and just as with high watermark storage volumes, reads attempted past the high watermark return zeroes.  


The next step up is the File Size. It is the VDL plus some extra pre-reserved space that has yet to be written to (uninitialized); also called the file tail. This is the full logical size of the file, shown as the combination of blue and green in the diagram, and is terminated by EndOfFile (EOF) flag. 

Lastly there is the Allocation Size, which indicates the full physical size of the file, and is comprised of the VDL and its following reserved space, up to the last cluster the file occupies any part of (may be some cluster slack). It is shown as the combination of blue, green, and red in the diagram. 

To aid in writing new data, the NTFS file system driver maintains a list of the largest free spaces on the volume (i.e. the starting LCN and run length). When a file gets created, it gets created in the free space that most closely matches the size of data available to write, in other words a "best fit". Additionally, a presumption is made that a newly created file will end up larger than the size that is currently available for the operating system to write, and extra free space, an “over allocation”, is reserved for the file so as to minimize fragmentation (see Microsoft Knowledge Base article ID 228198). The presumption is that the file will be 2, 4, 8 or 16 times larger than the currently known data size, depending on how much data is currently available for writing to the file in the operating system’s file cache. 

The file data is written to the volume, and the file is closed. Any over allocation is then released, returning to the free space pool and to the NTFS file system driver, if it qualifies as one of the largest free spaces on the volume. For this part, and this is a critical point, NTFS is very thin-friendly as when it reserves that over allocation, it can do so without writing to the volume (i.e. writing out zeroes). 

All said, this process does not eliminate fragmentation by any stretch and hence the continuing necessity to defragment the file system.   

One issue that does exist with NTFS, that presents universal challenges for thin provisioned storage, is the ability to recover space previously occupied by deleted files.  

This is an issue because when files are deleted in NTFS, the file system simply updates its metadata to indicate that the space occupied can be re-used for new file writes. A deleted file is not actually removed/wiped from the volume. Therefore, abstracted storage layers residing underneath NTFS may not be informed about this now, newly available free space. 

This creates a problem for thin provisioned storage which, if presented with limitations on re-use of space, could eventually exhaust all storage in the available pool. 

A solution for this challenge, commonly known as Thin Reclamation, encompasses the awareness of space formerly occupied by deleted data and actions then undertaken to recover and re-provision that space. There are a variety of solutions available to aid with thin reclamation such as zeroing deleted clusters to the SCSI UNMAP / SCSI WRITE_SAME commands, and will vary from vendor to vendor. 

Defragmentation and Thin Provisioning 

As covered earlier, defragmentation is vital to achieve and maintain peak performance. When Thin Provisioning is implemented on a shared virtualization host file system, it creates a high degree of probability of thin/dynamic virtual disk files themselves becoming fragmented, adding additional I/O overhead. In those storage systems, solving fragmentation becomes even more important.

However, for all the benefits of defragmentation, it is important to be aware of potential side effects. The side effects from defragmentation can vary from one thin technology implementation to the next, so it is important to know how the two technologies interact. 

Using special IOCTLs (I/O controls) in Windows, defragmentation is essentially moving data to consolidate file fragments and to pool free space into large contiguous extents. 

Where the provisioning technology allocates space on new writes, a defragmentation process (which is actually only moving data) will appear as new writes. Additionally, the former locations of moved data will not necessarily be known to be re-usable. Defrag will therefore generate additional storage capacity requirements for every piece of data moved. 

What can occur is that the new writes, are redundantly provisioned, which results in unnecessarily consumed space.  

Thin reclamation can effectively recover the wasted space, as could executing a data deduplication process (which would recognize and remove redundant data). 

Where high watermark provisioning is used, the watermark always increases and never decreases (on Windows), indicating less available space, creating a potential problem. If a file is written (or moved via defragmentation) to a higher cluster, the thin provisioning technology will need to provision space to accommodate. That is true even if the file is only moved to a high cluster temporarily.

On the opposite end of the spectrum, moving files “forward” can allow for space reclamation processes to better recover over provisioned space (depicted below).  

The process of compacting files to the front of a volume is something defragmenters can assist with.  

Proactive Fragmentation Prevention 

It is important to evaluate marketing claims from defragmentation vendors about “eliminating/preventing most fragmentation before it happens”; as the technology behind the marketing claim can have differing consequences for thin provisioned storage. 

Reactive solutions that rely on aggressive “free space consolidation” (packing files together) in order to rely on NTFS’es native “best fit” attempts will cause thin provisioned growth. 

Proactive technologies that do not require additional movement of any data in order to accomplish their objective do not cause increases in thin provisioned storage. They provide the benefit of a largely fragment-free OS file system without any negative consequences for thin provisioned storage. 

Patent pending IntelliWrite® technology, from Diskeeper Corporation, is such a proactive solution. IntelliWrite is a superior design (to NTFS native over-allocations) for reserving space at the tail of a file’s valid data. IntelliWrite is smarter in that it looks at the source of file writes/modifications and learns their behaviors over time. This heuristic process means that IntelliWrite knows better how much reservation space an open file needs to prevent fragmentation. It may be the file needs more than NTFS would natively offer, or it may pad less. The result of IntelliWrite’s intelligent over-allocations is an unmatched degree of successful fragmentation prevention (up to 85% success rate and more).   

Best Practices 

+Use proactive fragmentation prevention technology, such as IntelliWrite from Diskeeper Corporation.

+Know, from your vendor of choice, how they thin provision and what solutions they have for space (thin) reclamation.

+In Thin-on-Thin provisioned environments, space reclamation at one layer (e.g. thin virtual disk) does not necessarily address other provisioned storage on subsequent layers (e.g. LUN).  

+Defragment thin provisioned volumes when the corresponding storage growth can be addressed (e.g. a de-duplication/thin reclamation process)


+For high watermark provisioning, use a defragmenter that moves files to lower LCNs (i.e. the “front”). TVE and Titan Defrag Engines in Diskeeper and V-locity are designed to generally move files "forward".

+Use an OS/GOS defragmenter, or a defragmenter-mode that focuses on performance and not a “pretty” display.

+Apply SAN/VM vendor tools to eliminate fragmentation per their recommended practices for their proprietary clustered file systems.


+File sequencing/ordering technologies found in enterprise OS defragmenters can be quite valuable in many environments, especially performance-focused solutions on Direct Attached Storage. However, they can cause thin provisioned storage technologies to grow excessively due to their extra movement of data, so the general recommendation is to disable them or run them only when the effects (i.e. storage growth) can be addressed.



Want the full report? Download it from here: Best Practices for Defragmenting Thin Provisioned Storage.pdf (263.17 kb)

Tags: , , , , , , , , ,


Comment RSS

Month List


<<  February 2020  >>

View posts in large calendar