Condusiv Technologies Blog

Condusiv Technologies Blog

Blogging @Condusiv

The Condusiv blog shares insight into the issues surrounding system and application performance—and how I/O optimization software is breaking new ground in solving those issues.

not DEad yet?

by Michael 2. January 2011 16:31

"The reports of my DE-mise have been greatly exaggerated." (or something like this) 

-Mark Twain

First off let me state this blog on Drive Extender is entirely based on rumor. While I have significant access to resources at Microsoft, none of the following statements are based on info from any person(s) I've talked about in any prior blog posts.

I do have other sources...

That source affirmed that Microsoft realizes future possibilities for DE (local/cloud storage hybrid), in a broader scale than it is used today. 

With Google Chrome OS having generated some significant buzz on the matter recently, perhaps we'll see DE revived in the future after all?

unVAILed; an unfortunate DE-mise

by Michael 24. November 2010 09:51

If you're a Windows Home Server enthusiast, you've no doubt heard the unfortunate news that Drive Extender (DE) is being removed from the upcoming new version of WHS (codename Vail).

DEv2 was, by all accounts, an ambitious project, designed to solve many storage problems for numerous Windows platforms (Aurora, Vail, Breckenridge).  We'd had written about a technology partnership planned to assist with "garbage collection" for DE. The need to call in Diskeeper Corporation, I believe, was an indicator of the ultimate cause for its cancellation. Apart from time/resources, a.k.a. "money", there is no reason MS could not have solved the problem we were asked to help with, or any other problems for that matter.

The talent on the WHS team was certainly there, we know them well. The Technical PM for DE was a seventeen year vet at Diskeeper Corporation having formerly been the PM here for Undelete. A key consultant on the project was a file system guru who had worked with Diskeeper in the 1990's to implement kernel level code into Windows to make defrag, as we know it today, possible. We've had numerous meetings over the past many months and knew their challenges well.

WARNING! Objects in mirror are closer farther that they appear:

In science and technology, the end product often looks closer at the outset that it really is. You can only predict so much at the start. As you move along in research and development, many unknowns crop up that you have to understand and address. When you do finally reach the end of the road and look back, you realize you had to go a lot farther to get to the finish that you expected.

One case in point is the reason DEv2 used separate drive letters for each share. That was a solution to ensure that Windows Search would function. Windows Search does not work on "mounted volumes", which would be a superior user experience to assigning separate letters to each volume/share. Could Windows Search have been enhanced to support mounted volumes - yes (it is just software after all). However, that is another team at MS.

Now if WHS revenue was flooding billions in the MS bank account, Windows Search limitations would have been resolved lickity-split. As Paul Thurrott noted in an article yesterday, WHS is a niche product, that probably doesn't generate tremendous revenue for Microsoft. We developed Diskeeper HomeServer edition because we here love WHS (the company founder and top technical execs are all avid users), but mostly because we are committed to Microsoft's success - which is ultimately good for our business.

Niche products and pilot projects tend to be sandboxed and rarely get access to resources from other division within a company - and generally that is the right thing to do.  

At the end of the day, I believe (just my personal opinion) that the WHS team simply had to develop too many features for too many purposes and had to invent too many workarounds to make those features work. The decision to cut features and make a deadline is the most difficult one for software managers.

Anyway, I hold out hope that we'll see DE/DEv2 in the future. DE itself can be a solution foundation for future of Windows storage. Just one thought that crosses my mind is "what if DE merged local and cloud storage?" A user enabling folder duplication, if cloud storage services were activated, could replicate right into the cloud without the need for another layer of backup products/staging?

There is still significant value in Vail. The thing that really distinguishes WHS from NAS devices is that WHS is a platform whereas NAS is an appliance. Appliances will do next year the same thing they do today - which may work for many. A platform, through new/added software, can be customized to be a great variety of things. You build onto the platform whatever you want it to do for you, and as you grow, it can grow with you.  Because of that potential, it's amassed a base of passionate users with very high expectations (rightfully so).  

One cool thing about WHS is the ecosystem that has evolved around it. I fully expect that any shortcomings in Vail will be addressed soon-enough in after market solutions, so I'll wait before I pass judgment on Vail and decisions to upgrade.

Tags: , , , , ,


WHS "Vail" storage provisioning

by Michael 13. May 2010 06:15

As a Gold Partner, the first commercial defragmenter designed for WHS, and even one of the first 8 products to declare support for WHS when Bill Gates first announced it at CES several years back, we’ve been real close with the development team at MS.


The MS team’s insight into the needs and capabilities of defrag has led to significant cooperation as they complete the new Drive Extender. We’ve had many calls over the past months and are very happy to say that many of our suggestions and requests have been accommodated. The functionality that has been added from those meetings will result in well integrated and functional third party software solutions.


Supporting Drive Extender 2.0


Vail, which is in public beta, has already been generating a great deal of buzz on WHS forums. Drive Extender (DE) is a storage subsystem that extends storage functionality above and beyond what a typical Windows NTFS volume offers. Key purposes of DE are to offer fully automated and easy to use storage. All the complexities associated with fault tolerance solutions like RAID to provide drive reliability, expanding storage over time, and even solving data reliability concerns of commodity drives.


Drive Extender in Windows Home Server today implements drives independently and pools them into a common volume. This pool of storage is then presented as a single volume to the user (i.e. D:\). And, just off the root of this pool (D:\shares) you had all your WHS shared folders; e.g. Users, Photos, Videos, etc…


The user experience of WHS today is already such that you don’t need to care or interact with the volumes, you could even argue that its discouraged.


What is unique in DE 2.0 is that this paradigm has kind of been flip flopped. While it all still looks like a common repository the delineation of storage now begins with those shared folders. So, as an example, let’s take the shared folder “Photos”. In DE 2.0 that folder now becomes a dedicated NTFS volume presented out of the shared storage pool. The folder “Videos” becomes its own NTFS volume, and so on. This design was introduced to support features like real time folder level duplication, etc.. The only minor side effect is that because there are only 26 letters in the English Alphabet, there will be a limitation of the shared folders you can create in this location on WHS. Not a big deal, given the value of the features that this new design offers.


If you’ve read this blog and the comments, you’ll pick up that DE is extending a volume (i.e. a shared folder), in 1 GB chunks, taking those chunks from the total available storage pool.  


What you effectively have with Drive Extender, then and now, is storage virtualization. Any time you pool storage and then divvy it up exclusively to requestors (in this case the shared folders that become lettered volumes) you need some form of logic for allocating data from the pool. SAN and virtualization administrators already understand this concept, including related technologies such as Thin Provisioning.  

DE 2.0 now adds this to their storage virtualization solution. As you add more data to a shared folder, DE 2.0 will allocate, in 1 GB chunks, more space to the shared folder/volume from the common storage pool. And, should you delete ALL the data in a 1 GB chunk, the 1 GB chunk will dynamically return to the available storage pool to be allocated to any other shared folder that may need the space. DE is well designed to fill up 1 GB chunk before requesting to use more. Very cool stuff!


Here’s a demo of how the provisioning works. Assuming 20GB of space (divided into those 1GB chunks)



You now start to fill up storage adding a little over 4GB of photos and a little over 2GB of music files. That has now pulled eight 1GB chunks from the common pool and these volumes have dynamically expanded to hold up to 5 GB and 3 GB respectively. Keep in mind that files place in the Photos folder will NOT reside on the same 1 GB chunk that contains Music files. In this case, under the DE “covers” they are on completely separate Windows volumes.


And, as those eight 1 GB chunks are provisioned to shared folder volumes, the storage pool shrinks by 8GB.



Now… If you delete all the MP3 music files that reside in one of those 1GB chunks…


DE can return that chunk back to the storage pool for re-provisioning re-use with any other shared folder.



Subsequently shrinking the Music folder/volume to two 1 GB chunks:



However, there are some conditions in which this provisioning technique can use some assistance, and Diskeeper will be helping out (per the request of the WHS/DE team at MS). Should you delete some of the data from 1 GB chunks, but not all of the data within a 1 GB chunk, you can have a lot of 1 GB chunks allocated to a shared folder/volume, but not actually using all of the space it is taking from the common storage pool. Diskeeper will be helping in these cases to group together all the data spread across sparsely filled 1 GB chunks. We’ll effectively be squishing the data together aligning it along 1 GB boundaries. The benefit of this is that some 1 GB chunks may then be freed up and returned to the storage pool to be assigned to your other shared folders.


Here’s a quick graphic to explain the process. Five 1 GB chunks are taken up by the Photos folder/volume. Over time, unwanted photos may be deleted, but the space they were taking up is not made available to any folder other than Photos. In order to make the space (3GB in this example) available for Videos or Music, you would need to move the data out of the sparsely filled chunks. Once done, those now empty 1 GB chunks can be used elsewhere.



Is this an issue you’ll come across? Maybe. If you do, it’ll likely be a bit of time and a lot of file deletions down the road. The Microsoft provisioning design is well suited to most users who mainly add and retain data. Those who do housecleaning or more involved data management can benefit from the upcoming Diskeeper solution – stay tuned.


While we’d like to take credit for this new feature in a future release of Diskeeper HomeServer, it was quite frankly MS directly asking us, as a partner, to add this into our product. Perhaps they’ll add this data squishing into WHS down the road? In the mean time, you can look to Diskeeper to help.


PS: thanks to the Microsoft WHS team for reviewing and approving this blog post.



Diskeeper | WHS


Comment RSS

Month List


<<  January 2020  >>

View posts in large calendar