smartctl and smartd: Catch Failing Disks Early

"SMART overall-health self-assessment test result: PASSED" is the most over-trusted line in server operations. A drive can report PASSED while reallocating sectors every hour, and it can flip to FAILED the same day it stops answering. The value of SMART is in the trend of a handful of attributes, plus scheduled self-tests and a daemon that tells you early. Here is the workflow that actually prevents data loss instead of documenting it.

The Three Commands You Need First

smartctl -H /dev/sda            # health summary (the one-line answer)
smartctl -a /dev/sda            # identity + attributes + error log
smartctl -x /dev/sda            # everything, including vendor-specific data

# NVMe devices
smartctl -a /dev/nvme0
nvme smart-log /dev/nvme0       # alternative with nvme-cli

Remember the RAID case: behind a hardware controller, /dev/sda is the virtual disk. Query physical drives with smartctl -a -d megaraid,N /dev/sdX, and use smartctl --scan to see what smartmontools can reach. USB bridges often need an explicit -d sat.

Attributes Worth Watching

Copy the full attribute table to your monitoring system, not just the health verdict. A script that exports the CSV of all attributes once a day turns "we had no warning" into a visible slope on a graph.

Self-Tests: Short, Long, and Scheduled

smartctl -t short /dev/sda        # ~1-2 minutes, catches gross issues
smartctl -t long /dev/sda         # full surface scan, hours — run off-peak
smartctl -l selftest /dev/sda     # results of past tests
smartctl -l error /dev/sda        # error log

A short test is fast enough to run weekly on every drive. The long test is the one that finds failing areas on spinning media; schedule it so it does not overlap with backup windows, and expect a performance dip while it runs.

smartd: The Daemon That Tells You First

# /etc/smartd.conf
# monitor all ATA/SATA/SCSI/NVMe devices, run tests, email on problems
DEVICESCAN -a -o on -S on -n standby,q -s (S/../.././02|L/../../7/06) \
           -m admin@example.com -M exec /usr/libexec/smartmontools/smartd-runner

# or monitor specific devices by UUID
/dev/disk/by-uuid/820cdd8a-866a-444d-833c-1edb0f4becac -a -m admin@example.com

Wiring -M exec into your monitoring system beats email: a drive degradation alert should land in the same place as the rest of your infrastructure alerts, with the attribute values attached.

Replacing a Drive Without an Outage

# Linux software RAID: force a copy to a spare instead of a rebuild from parity
mdadm /dev/md0 --add /dev/sdd1 --replace /dev/sda1

# ZFS
zpool replace tank /dev/sda /dev/sdd

# Check the replacement took over and the old device is gone
cat /proc/mdstat ; mdadm --detail /dev/md0
zpool status -v tank

Order of operations that saves time: identify whether the failure is the disk or the path (CRC errors point at cabling), open the warranty case before the drive is fully dead (vendors want the SMART log), and replace proactively on a rising reallocation count rather than waiting for FAILED.

Where SMART Lies to You

• Virtual disks and some USB enclosures pass through a fabricated or empty SMART table — do not build alerting on it.
• PASSED is not a health guarantee: a drive with pending sectors can still report PASSED. Trend the attributes.
• Behind RAID controllers the OS-level device is not the physical drive; scan with -d megaraid,N or use the vendor tool.
• One cable, many CRC errors: do not replace a drive for attribute 199.

相关阅读：RAID 级别与写洞解析、Ceph OSD 掉线与 PG 排障 以及 LVM 精简置备与精简快照。

原文链接：ArchWiki - S.M.A.R.T.