RAID Levels, Parity and the Write Hole Explained

RAID levels are taught as capacity and redundancy arithmetic, but the operational decisions hinge on two things the tables rarely mention: rebuild time with large drives, and the write hole in parity arrays. This article compares the levels that matter in production, then explains the write hole and the journal that closes it on Linux md arrays.

The reason RAID tables are misleading in practice is that they optimise for the wrong variable. A RAID 5 array looks attractive on paper - you lose one disk's worth of capacity and gain single-disk protection - but on multi-terabyte drives the rebuild is measured in hours to days, and every read during that window is a chance to hit a latent media error on a survivng disk. Meanwhile RAID 10 gives up half the capacity but rebuilds in a fraction of the time and keeps serving reads from the surviving mirror. The interesting decisions are about exposure windows, not raw arithmetic.

The Levels That Actually Ship

• RAID 0 - striping only. Highest throughput, zero protection: one disk failure loses the set. Fine for scratch data, never for anything you would restore.
• RAID 1 - mirroring. Survives one disk per mirror, halves usable capacity, and read performance scales with the number of mirrors. Write cost is doubled unless the controller caches.
• RAID 5 - striping with single distributed parity. n-1 usable, survives one failure. Rebuilds read every surviving disk; with today's multi-terabyte drives the rebuild window is measured in days, and an unrecoverable read during the rebuild costs the array.
• RAID 6 - striping with dual parity. n-2 usable, survives two failures and, critically, survives one failure plus one read error during rebuild. This is why large-capacity SAS/SATA arrays standardised on RAID 6.
• RAID 10 - mirror then stripe. Half the capacity, but fast rebuilds (copy one mirror, not recompute parity) and far lower exposure during a degraded window.

Rule of thumb: if the array is made of large spinning disks and the data matters, RAID 6 or RAID 10. If write latency matters and capacity is affordable, RAID 10.

Choosing between the two is a workload question. RAID 10 wins where write latency and small random I/O dominate - databases, virtual machine stores, anything with a heavy metadata or commit workload - because there is no parity calculation and reads can be spread across both halves of each mirror. RAID 6 wins where the workload is large sequential I/O or capacity is expensive and nearline performance is acceptable - backup targets, media archives, bulk file storage. Neither is "safer" in the abstract; RAID 10 protects you during the degraded window better, while RAID 6 protects you against two simultaneous failures, which is the more likely hazard once rebuilds stretch into days.

The math behind rebuild risk is worth internalising. As drive capacity grows, so does the time spent reading the surviving members, and enterprise SATA drives are specified with a bit error rate that translates to roughly one unrecoverable read error per 10^14 to 10^15 bits, i.e. one read error per 12TB to 125TB read. Rebuilding a RAID 5 array of 8TB drives means reading several times 8TB - so the cumulative probability of encountering an unrecoverable read during the rebuild climbs into the tens of percent. Under RAID 6, that same read error during the rebuild is survivable. That single fact, not peak throughput, is what ended RAID 5 in large arrays.

Also note what RAID cannot give you: it protects against disk failure, not against controller failure, firmware bugs, accidental deletion, or site disaster. Layout metadata matters too, so record stripe size, chunk size and member order somewhere outside the array - an array whose exact geometry is unknown turns a rebuild into a forensic recovery project.

What the Write Hole Is

In RAID 4/5/6 a stripe write touches several disks. If the system loses power before all data and parity blocks land, the stripe is left inconsistent: data and parity no longer agree. The array is marked degraded and md resynchronises - but until that resync completes, a crash during the rebuild can expose the inconsistent stripe and corrupt real data. That is the write hole, and it exists in any parity RAID implementation without a journal or a battery/flash-backed write cache.

The subtle part is the timing. The write hole is not a risk only at the moment of the first power loss; it is a window that stays open until the array has been fully resynchronised. While the array is degraded and being rebuilt, its parity is incomplete, and the reconstruct-on-read logic that would normally rebuild missing data relies on that incomplete parity. A power loss inside that window - a second power cut, a PSU replacement, a rack PDU trip - can write a bad stripe back, and the corruption is baked in. Software RAID without a journal or a hardware RAID without a non-volatile write cache both expose this window.

Closing the Hole with the Linux md Journal

mdadm --create /dev/md0 --level=5 --raid-devices=4 \
  --write-journal /dev/nvme0n1p1 /dev/sd[b-e]1

cat /sys/block/md0/md/journal_mode
echo write-back > /sys/block/md0/md/journal_mode
echo write-through > /sys/block/md0/md/journal_mode

The cache disk sits alongside the RAID members and absorbs writes first. In write-through mode md calculates parity, writes data plus parity to the journal, then flushes to the members and only then reports completion - the journal disk does not need to be large, and cache-disk failure only reopens the write hole rather than losing data. In write-back mode writes complete as soon as they are in the journal, which also speeds up non-full-stripe writes; the trade-off is that reads may have to query the journal and the cache disk becomes a read-path dependency.

The write journal works because writes are ordered: the journal is a dedicated device that records the intent to write a stripe before the stripe itself is touched. After a crash, md replays the journal on assembly, completing or rolling back the interrupted stripe so the array never carries half-written parity. The choice between modes is a latency-versus-dependency trade. Write-through keeps the members authoritative and treats the journal as a safety log, so it is the conservative choice for data you cannot regenerate; write-back is faster and better for random non-full-stripe writes, but you must accept that the journal device is now on the read path and that its loss degrades you. Whichever mode you choose, put the journal on media that is genuinely more reliable than the members - a mirrored NVMe partition, or an SSD that reports its own health - and monitor it like a member rather than treating it as disposable.

Hardware arrays solve the same problem differently: a battery- or capacitor-backed write cache (BBWC/FBWC) holds pending writes through a power loss and flushes them on restore, which closes the hole at the controller level. If you run hardware RAID without a protected cache, you have the write hole whether or not the vendor's datasheet mentions it. For a software stack that also needs to survive a wider failure, remember the journal does nothing about the data itself - replication and snapshots are a separate layer.

Building the array without regrets

mdadm --create /dev/md0 --level=6 --raid-devices=6 \
  --chunk=512K --bitmap=internal /dev/sd[b-g]1

mdadm --detail --scan >> /etc/mdadm.conf
echo check > /sys/block/md0/md/sync_action

Three parameters decide how the array behaves for its whole life. Chunk size sets how much contiguous data md writes to one member before moving on: small chunks spread load across disks and suit random I/O, while large chunks (512K and up) help large sequential reads and writes. Layout for RAID 6 controls how the two parity blocks are placed, and it must be consistent across members or the array will not assemble. The write-intent bitmap records which regions are dirty so that after a crash md resynchronises only those regions instead of the whole array - it costs a little write throughput but turns a multi-hour resync into a minutes-long one, which directly shrinks the degraded window. Write the assembled config to /etc/mdadm.conf and rebuild the initramfs so the array comes back automatically after a reboot; an array that fails to assemble at boot is a self-inflicted outage.

Finally, give the array a hot spare so a replacement starts rebuilding without waiting for a human:

mdadm /dev/md0 --add /dev/sdh1
cat /proc/mdstat

A hot spare does not remove the need to monitor - a second failure while the spare is rebuilding is still a risk - but it removes the minutes or hours between "someone notices" and "rebuild begins", which is where most large arrays are actually lost.

Operational Checklist

• Verify the array after creation, and schedule patrol reads so latent media errors surface before a rebuild does.
• Monitor the parity-initialisation/resync progress and never assume an array is protected until it completes.
• Keep a spare disk (or hot spare) per shelf, and record the exact layout - a lost array with unknown stripe order is a recovery project.
• For SAN arrays, the equivalent protection is a battery or capacitor-backed write cache plus periodic scrub; the vendor-side disk replacement workflow is described in HPE 3PAR ServiceMag disk replacement and array-level resiliency in Dell PowerStore best practices.

Watch rebuild progress explicitly rather than inferring it from the array feeling fine:

cat /proc/mdstat
mdadm --detail /dev/md0
watch -n 60 'cat /proc/mdstat'

A freshly created array spends a long time in resync or recovery, and it is not protected until that finishes. A mismatch_cnt that keeps climbing after a scrub is a warning sign of media or cabling trouble and should be investigated, not ignored. RAID is not backup: snapshots and replicas protect against the failures parity cannot, as covered in ONTAP snapshot policies.

原文链接：https://www.kernel.org/doc/html/latest/driver-api/md/raid5-cache.html