Robust Design: Fault Tolerance – Performance vs. Protection

Monday, May 24th, 2010 by Robert Cravotta

[Editor's Note: This was originally posted on the Embedded Master

Fault tolerant design focuses on keeping the system running or safe in spite of failures usually through the use of independent and redundant resources in the system. Implementing redundant resources does not mean that designers must duplicate all of the system components to gain the benefits of a fault tolerant design. To contain the system cost and design complexity, often only the critical subsystems or components are implemented with redundant components. Today we will look at fault tolerance for data storage on long term storage, such as hard disk drives used for network storage to illustrate some of the performance vs. fault tolerance protection trade-offs that designers must make.

As network storage needs continue to grow, the consequences of a hard disk drive failure increase. Hard disk drives are mechanical devices that will eventually fail. As network storage centers increase the number of hard drives in their center, the probability of a drive failure goes up in a corresponding fashion. To avoid loss of data from a hard drive failure, you either must be able to detect and replace the hard drive before it fails, or you must use a fault tolerant approach to compensate for the failure.

Using a RAID (redundant array of independent disks) configuration provides fault tolerance that keeps the data on those drives available and protects it from loss through a single disk drive failure. Using a RAID configuration does not protect the data from application failures or malicious users that cause the data to be overwritten or deleted; regular data backups are a common approach used to protect from those types of failures. RAID is a technique for configuring multiple hard disk drives into a single logical device to increase the data reliability and availability by storing the data redundantly across the drives.

A RAID configuration relies on a combination of up to three techniques: mirroring, striping, and error correction. Mirroring is the easiest method for allocating the redundant data and it consists of writing the same data to more than one disk. This approach can speed up the read speed because the system can read different data from different disks at the same time, but it may trade off write speed if the system must confirm the data is written correctly across all of the drives. Striping is more complicated to implement, and it consists of interlacing the data across more than one disk. This approach permits the system to complete reads and writes faster than performing the same function on a single hard drive. Error correction consists of writing redundant parity data either on a separate disk or striped across multiple disks. Storing the error correction parity data means the amount of usable storage is less than the total amount of raw storage on all of the drives.

The different combinations of these three techniques provide different performance and fault protection trade-offs. A system that implements only data striping (RAID 0) will benefit from faster read performance, but all of the data will be at risk because any disk failure will cause loss of the data in the array. Data recovery can be costly and not guaranteed with this approach. This approach is appropriate for fixed data or program structures, such as operating system images that do not change often and can be recovered by restoring the data image from a backup.

A system that implements only mirroring (RAID 1) creates two identical copies of the data on two different drives; this provides data protection from a drive failure, but it does so at the cost of a 50% storage efficiency because every bit of data is duplicated. This approach allows the system to keep a file system available at all times even when performing backups because it can declare a disk as inactive, perform a backup of that drive, and then rebuild the mirror.

A system that implements data striping across two or more data drives with a dedicated parity drive (RAID 3 and 4) provide better storage efficiency because the fault tolerance is implemented through a single extra drive in the array. The more drives in the array, the higher the storage efficiency. However, all reads and writes access the dedicated parity drive, so the dedicated drive throttles the maximum performance of the system. RAID 5, which stripes the parity data across all of the drives in the array, has all but replaced RAID 3 and 4 implementations.

RAID 5 offers good storage efficiency as the parity data consumes the equivalent of a single drive in the system. This approach suffers from poor write performance because the system must update the parity on each write. Because the parity data is striped across the drives, the system is able to continue degraded (slower) operation despite a hard drive failure. The system is able to rebuild a fault tolerant data image with a new disk drive, such as on a hot swap drive, while the system continues to provide read and write access to the existing data. RAID6 extends on RAID 5 by using additional parity blocks for dual fault tolerance.

Each of these configurations provides different performance and fault tolerance trade-offs and are appropriate based on the context of the subsystem they are used for. Note that a single RAID controller itself can be the single point of failure within a system.

Performance and protection trade-offs are not limited to disk storage. An embedded design might a firmware library from Flash to SRAM to gain a performance boost, but the code in SRAM is now vulnerable to unintended modification. If you would like to participate in a guest post, please contact me at Embedded Insights.

Tags:

Leave a Reply