Sleep easy

Hard disks are always going to fragment, and defragmenting them is usually a good idea – but in these days of fast drives and multi-level caches, it’s not such an issue and not worth getting worked up about.

File fragmentation keeps some people awake at night. When DOS was new, a colleague used to amuse me by always leaving his PC defragmenting its disk whenever he was away from his desk for more than a few minutes – as well as every night. I thought this obsessive, and possibly even damaging to the machine owing to the extra load on the hard disk. Today, fragmentation isn't such a high priority but it can still cause panic. So let's take a look at the state of play.

First, what exactly is fragmentation? As you use a hard disk, files become scattered over its surface. As they are deleted, the free space also becomes broken up into small areas. As a result, disk I/O slows down because the drive has to move its read/write head more than is strictly necessary. All disks gradually become fragmented, and therefore slower to access, the more they're used. Clearly, fragmentation is bad for a desktop machine and even worse for a server, but just how big a problem is it?

Stop and start
The key issue is disk head movement. When a file is written or read, the operating system moves the read/write head to the start of the free space or to the start of the file and begins transferring data. At this point everything happens quite quickly, but if for some reason the read/write has to be stopped and started at another physical location on the disk, the movement of the heads slows things down. In single-user/single-tasking systems, fragmentation is pretty much the only cause of unnecessary head movements. If a file is spread out over the disk, the head has to be moved to read it. Similarly, if the free space is fragmented into small blocks then writing a new file that is larger than one of the blocks involves additional head movement.

Fragmentation can also occur on units that are larger than a single file. For example, suppose an application always reads a particular set of files in a particular order. Files are often created in the order in which they will be used, thereby minimising head movement. However, as files are created, destroyed and rewritten, files that form logical groups tend to become separated on the disk. When this happens, the disk head has to move further to get from one file to another than would otherwise be necessary.

The purpose of defragmenting, or "defragging" as it is often called, is to reorganise the used areas of the disk so that every file, as well as the remaining free area, constitutes a single disk area that can be read from or written to with minimum disk head movement. Defragmentation works by moving files and portions of files around the disk. This is a dangerous operation because shuffling the contents of a disk invites disaster if the power, or anything else, fails during the operation. The Windows defragmentation tool is designed to make the process as safe as possible. Completely defragmenting a large disk can take quite a while, but you can manually interrupt the process. The files that have been defragmented won't revert to their fragmented state and overall you'll notice an improvement in the performance of the disk. What's more, when you restart the defragmentation there is less work to do – so even a little defragging is worthwhile.

No easy answer
The real issue, though, is not so much fragmentation as optimal disk layout. In other words, how should files and directories be physically laid out on a disk to maximise performance? The answer, of course, depends on the sequence of read/writes the disk will be subject to. A few applications, notably Microsoft Office, optimise file layout when you install them, and sophisticated utilities such as Diskeeper 2008's I-FAAST technology monitor file access and attempt to optimise the disk. But in most cases defragmentation is about as good as disk layout optimisation gets.

The problem becomes much more complicated in multitasking/multi-user systems. Even with a single user and an ideal disk layout, the ordered sequential access to files that characterises a single-tasking environment is replaced by something much closer to random access as the system switches from one task to another. What matters here is the frequency with which the computer switches between tasks, rather than the time taken to complete the average disk task. This suggests that fragmentation is less of a problem for servers than for workstations.

The big problem in deciding how important fragmentation is likely to be is that few useful studies exist, and most of the "best practice" advice is based on early experience with FAT filing systems based on MS-DOS. NTFS, which is or should be the filing system of choice, is much less susceptible to fragmentation. When you consider too that disk drives are now much faster than they were, and are generally accessed though multi-level caches, you can see that fragmentation isn't the problem it once was.

This isn't to say that you should ignore it altogether, but you needn't be obsessive about eliminating it. Modern file systems take longer to become badly defragmented, and as long as fragmentation isn't extreme (which generally doesn't happen unless a disk is nearly full), the increase in performance that you get from a full defragmentation probably isn't worth the extra CPU time involved. Of course, running a defragmenter will always produce some increase in speed, and is unlikely to do any harm, but do you really need to worry about decimal point changes in task timings?

Don't panic
So what should you do about fragmentation? The first thing is not to overreact – if a server is behaving well, don't rush to fix what isn't broken. You should certainly defragment a system when you first set up a new disk. This is a cheap and easy way to reduce subsequent fragmentation. Try to keep disk usage below 75 per cent as this also reduces fragmentation and makes it easier for a defragmenter to work. If you can schedule defragmentation to occur when the machine isn't under load, so much the better – but make sure you back up the data before starting the process.

In most cases it isn't worth worrying too much about completely defragmenting the free space on a drive. That's because NTFS rapidly fragments the free space in normal use and there isn't a great deal you can do about it. Even Microsoft admits that it's not worth trying to completely defragment free space. In fact, the latest defragmentation tool included with Vista and Windows 2008 doesn't bother to consolidate any fragment larger than 64MB – it's as if 64MB is a large enough chunk for any hard disk to process before having to move its heads for reasons other than fragmentation.

As always, there are exceptions to the rule. For example, any machine that's engaged in realtime work, such as video or audio encoding, must have a fully defragmented hard disk. In this case the task is fully sequential and any unnecessary movement of the disk heads can result in lost data. Even here, the problem is much reduced if the machine has lots of processor power and intermediate caches.

A second exception is any server with a high data "churn". Most servers don't frequently create and destroy data. Usually a document is created and then accessed many times before it is changed – think of a web server, for example. Some applications, though, do churn large numbers of documents or records. For these, the degree of churn has to be judged against the size of the drive, the amount of free space and the size of the average document – a difficult task.

All in moderation
I'm not claiming defragmentation shouldn't happen, or that it is no longer important. Running a defragmentation utility is a good idea, but you don't have to be obsessive about it unless the machine has to perform exceptionally or its storage is nearly full. And in the second case, providing extra storage is a much better solution than improving defragmentation.