Sky's the Limit

We all want to store increasing amounts of data and, as ever, storage is getting bigger, faster and cheaper. The real issue is whether we are making best use of it and paying enough attention to data organisation

The simple summary of where we are and where we are going is “bigger, faster, cheaper” but this is just the driving force. What is more interesting is what we are doing with all that fast, cheap storage and how we could be doing it differently.

For example, having so much more storage to handle might lead you to believe that inventing some new sort of backup device was an urgent need. In fact, the availability of low-cost hard disks makes it possible to think the, until quite recently, unthinkable idea of using more disks to back up disk storage. On the face of it, disk-to-disk (D2D) backup is a crazy idea in that hard disks aren’t removable and hence can’t be archived in the same way as tape or other removable media – but there are lots of situations where archiving is a secondary consideration. You could say that it’s this sort of synergy that makes IT more interesting than just more of the same.

Storage trends
So what trends in storage are likely to make us work differently in the coming months? Strangely, one of the growing concerns for everyone building new storage facilities is energy use. It’s all too easy to focus on the problem of server power consumption, but in reality there are usually more disk drives than processors in a typical setup. Server technologies have evolved to make it possible to do more with fewer, more energy-efficient processors. Next on the agenda, reducing the power used and the waste heat produced by hard disks is key.

Manufacturers of storage systems – NAS and SAN – are being quick to claim energy savings in their new systems, but in fact we probably should be thanking the drive manufacturers. Most of the system savings come from the hard drives used. For example, Hitachi has cut its latest drives to about 50 per cent of the energy usage of standard drives and currently 8W is a typical power consumption for a modern drive. What this means is that you could achieve a power saving simply by replacing drives with newer, low-power devices. It’s a bit like installing energy-saving light bulbs. However, if storage is going to emulate the savings that servers have made, we have to look to reducing the number of drives in use. At the moment this seems an unlikely target as the increases in efficiencies brought by increased reliability and data compression have their own serious problems.

For real power savings we have to turn to solid state disks (SSD). I discussed these a few months ago and progress in this area is still rapid. SSDs have their problems, but for many applications they are increasingly useful. Asus, for one, has used SSD storage to produce some remarkably small and energy-efficient PCs and it is only a matter of time before the same technology makes its way onto every desktop machine and server.

At this point will it really mark the end of the hard disk? This is an event so regularly predicted that it’s tempting fate to do so once again. The answer is, once again, probably not. Hard drives and magnetic devices in general seem to have a way of evolving each time the pressure is applied. In this case the cost per megabyte of storage for hard disks is likely to make them the media of choice for some time – not as secondary storage, but as a third-level system where total capacity rather than speed or efficiency is an issue. Yes, we still need RAID and similar systems and these will be providing the bulk of our storage needs for the medium-term future, but SSD will come to dominate the desktop and the portable world much more quickly than you might imagine.

Hard-diskless workstations have obvious advantages and focus the mind perfectly on the separation between desktop and enterprise-level storage systems. Who knows, in a reasonably short time the hard disk might become a specialist piece of equipment found only in enterprise storage systems – what effect on price and performance this might have is interesting to speculate on.

There’s no limit
While on the subject of large-scale storage, there seems to be no avoiding the fact that we have no limit on the data we seem capable of generating and passing around. With typical mailbox allocations rising to the gigabyte range and photos, movies and sound files becoming standard, the average user is moving towards a terabyte of personally generated data. No longer are such large data sets the preserve of company-wide database systems – every user will expect you to manage terabytes of data on their behalf.

Lost and found
This thought highlights what we always knew – storage is inefficient and difficult to use. A typical user isn’t going to organise their data and most of what they accumulate, valuable though it might be, is simply going to be lost in the sense that they won’t be able to find it when they want it. Worse, they probably won’t even remember that it exists and could serve some useful purpose. Try finding a suitable photo of a product to use in a sales brochure – in most cases it’s easier to commission a session with a photographer.

As data volume grows, finding, organising, archiving and generally working with it grows ever more difficult. Operating systems such as Vista have added features to make finding data easier, but these are still inadequate and a long way from the Windows Future Storage (Win FS) system that was to have been included. This proved too difficult to implement and so didn’t ship with Vista and hasn’t shipped with Windows 2008. Microsoft seems to be working on it still but, to be honest, we almost certainly need a Win FS version 2 by this time. One thing that is certain is that whatever the future filing system is, it will be heavily object-oriented and we need to prepare to be ready to deal with object stores rather than just data stores.

Currently there is no solution to this problem and no convincing candidate on the horizon. We need to invent something that organises and compresses data to make it easier to look after and easier to access.

The current buzzword is “deduplication”, or “deduping”, but when you consider that this is just the application of traditional compression techniques, similar to those used to create Zip files, it looks less revolutionary. It may be very useful, but it’s just a start on the bigger problem.

Safety first
Clearly, to meet the ever-increasing need to keep all of this data safe, we have to be looking at the growth of specialised archival storage facilities and this brings us on to the issues of the Web, virtualisation, online storage and many other topics.

Online storage is a growth area with companies such as Microsoft (SkyDrive), Amazon and Google all keen to develop the idea. Online storage has so many advantages – it can be distributed, it provides off-site archiving, you can pass the buck and have someone else look after your data, it’s accessible from any location and by mobile users just as easily as fixed users – that it is sobering to recall what the main problem is. Online data storage suffers from the bottleneck of local connectivity.

It is still not really practical to transfer your data from local to online storage using the typical speeds offered by the Internet, even if you are prepared to pay to do so. The only real alternative is to use clever incremental data transfer algorithms that essentially treat the online data store as an incremental backup site. This works, and it works well for small amounts of data, but connectivity begins to be the limiting factor as soon as the volume increases. Even so, it is probably the way of the future.

Way to go
Microsoft’s research into Midori – the code name for a future version of Windows – reveals that it thinks that a distributed implementation using the currently popular ideas of cloud computing and Software as a Service are the way to go.

Of course, part and parcel of this SOA, messaging-based approach is the idea of online storage. Future operating systems could push the move to online storage that began with the Web into the areas normally reserved for desktop application and their associated desktop storage. Without the need to store operating systems and applications locally, this even makes the concept of the diskless workstation, or better the small SSD-based workstation, much more natural.

What is the future? The answer is always bigger, cheaper, faster… but this really just covers up the real answer. At the moment we struggle with the apparently trivial problems of creating reliable, available and manageable storage systems and leave the real problems of data organisation and retrieval mostly to the end user and this has always been a recipe for trouble.