Clustered storage for surveillance

As organisations’ video surveillance requirements become more complex, storage requirements increase. The digital video surveillance market has some unique requirements, and a new form of clustered storage offers benefits.

Clustered storage can be especially helpful when upgrading from small-scale recording environments to large-scale surveillance systems that support thousands of cameras and petabytes of storage.

Today, whether we are dealing with ‘normal’ business data (emails, files, database transactions, etc.) or video data (dense video images that are constantly streaming into the storage system), storage systems deploy RAID to protect against any disk drive failures in the field. The way in which these RAID controllers are designed can have a dramatic effect on the performance, cost, and manageability of the systems. Both business and video data have marked characteristics and large ramifications for storage systems.

Data Set Size: how much raw data needs to be stored?

A large database today is roughly 100 gigabytes and only very large companies create one terabyte (1,000 gigabytes) of general business data. By contrast, a video surveillance system with just a few hundred cameras can easily generate more than one terabyte of data in a single day.

Read vs. write activity

Normal business data, whether it is payroll database entries, legal file updates, or email messages, generally follows the 80/20 rule, where 80% of the time data is being read and only 20% of the time is dedicated to write activity. Video data has nearly the exact opposite characteristics, with data being written 100% of the time and read activity happening only rarely when incidents occur and the relevant footage needs to be retrieved and reviewed.

Random vs. sequential

A storage administrator would look at how data is sent to the storage system and how this handles that traffic. In the normal business case, data is written randomly in short bursts in what is termed “small block transfers”. Video data, once again, is radically different as it arrives in a sequential and constant stream of dense information. As camera resolutions increase, this density grows geometrically as more pixels are used to convey each image.

Planned downtime

Business data generally follows a cycle with a natural rhythm of downtime when maintenance, upgrades, and changes can be made to the system without affecting business activity. Video data has no such downtime cycle since the incoming stream of video is unrelenting and any changes or maintenance must be made dynamically without affecting the incoming stream.

A traditional large-scale storage system is designed to accommodate business data. The architecture of these systems is well-established and is centred around the concept of a master RAID controller with a fixed amount of network bandwidth that controls a fixed maximum number of drives. Traditionally, this master RAID controller has acted as a funnel for incoming video data and limited the maximum capacity of the system. More advanced technology has introduced a secondary controller for redundancy, but the maximum bandwidth and capacity remain fixed.

Video data presents quite a challenge to this traditional storage setup. The funnel design, which is rarely overwhelmed for business data, is poorly suited for the 100% write and sequential nature of video data and introduces data loss risks. Traditional designs tend to need either next-generation networks such as 10 Gigabit Ethernet, or proprietary, expensive storage protocols such as Fibre Channel, in order to meet the bandwidth requirements of large-scale video installations. Similarly, the master controller architecture is problematic for field maintenance since capacity, bandwidth, and performance changes are difficult, if not impossible, to dynamically upgrade in the field.

Newer clustered storage architectures aggregate a set of distributed RAID controllers to provide large-capacity storage. While providing the same data protection as traditional designs, the clustered systems introduce a number of innovations that are particularly useful for video. For example, data is spread automatically across the cluster, performance is allocated dynamically without user intervention, and the cluster manages failure conditions for drives, network connections, and complete controllers. In storage terms, these features are termed dynamic provisioning, automatic load balancing, and multi-pathing support.

In the best cluster implementations, the bandwidth of all the clustered RAID controllers is aggregated together and available to all the incoming video data, thereby removing the traditional funnel limitation using the cost-effective Gigabit Ethernet technology available today.

For video, this is the optimal architecture for providing the high-bandwidth that is essential for the write-mostly, sequential nature of this type of data traffic. Clustering provides high bandwidth by aggregating several network links, and the architecture can deploy off-the-shelf, Gigabit Ethernet switches and cables for low installation and support costs. Dynamic field upgrades are easily maintained by adding further clustered components. A side benefit of the architecture is that automated system rebuilds as in the case of a hard disk replacement, are accelerated over traditional designs because all system components contribute to a massively parallel recovery.

Next page: What to look for in storage for surveillance