Ceph OSD Down and Stuck PGs: A Triage Runbook

Ceph alarms read worse than they are. A single down OSD in a three-replica pool is a warning, not an outage; an incomplete placement group is a data-loss event. The skill is knowing which of the two you are looking at within the first thirty seconds. This runbook gives the triage sequence, what each PG state means, and the safe procedure for replacing an OSD without triggering a rebalance storm.

Three commands answer 90 % of Ceph questions

ceph -s
ceph health detail
ceph osd tree
ceph pg stat
ceph df detail

Start with ceph -s: it tells you the health code, the number of down OSDs and the PG distribution. Then ceph health detail names the actual objects of concern — which PGs are stale, which are incomplete, and which OSDs are down. Only then look at the tree.

Down OSD: find out why before restarting

ceph osd tree | grep down
ssh pve-XX
systemctl status ceph-osd@7
journalctl -u ceph-osd@7 --since '1 hour ago'
dmesg | grep -i -E 'I/O error|medium error'
smartctl -a /dev/sdX | head -40

If the journal shows I/O errors, an OOM kill or a hardware fault, restarting the daemon will only delay the inevitable. If it shows a clean stop (a manual stop, a failed upgrade), systemctl start ceph-osd@7 brings it back and Ceph recovers automatically. Let recovery finish before you touch anything else — backfilling while a second OSD is flapping is how a warning turns into HEALTH_ERR.

Placement group states and what they mean

• active+clean — healthy; most PGs should be here.
• active+degraded / active+remapped — recovery or rebalancing in progress; normal after an OSD change.
• stale — the monitors have not heard from the PG’s primary OSD (governed by mon_osd_report_timeout); usually all replicas are down.
• down — the PG has no primary that can serve; the OSD holding the authoritative copy is out.
• peering — OSDs are negotiating which copy is authoritative; transient, but a stuck peering state points at a connectivity or version problem.
• incomplete — objects are missing from all replicas. Treat this as real data loss and stop making changes.

ceph pg dump_stuck inactive
ceph pg dump_stuck unclean
ceph pg dump_stuck stale
ceph pg map 1.4f
ceph pg deep-scrub 1.4f

ceph pg map <pgid> shows the acting and up sets. When they differ, Ceph is mid-rebalance; when they match and the PG is still stuck, the problem is on the OSDs themselves.

Replacing a failed OSD safely

1. Mark it out, never destroy first: ceph osd out osd.7, then wait for active+clean.
2. Confirm nothing regressed: ceph -s should show no degraded PGs before you continue.
3. Destroy and zap: ceph osd destroy 7 --yes-i-really-mean-it, then ceph-volume lvm zap /dev/sdX --destroy.
4. Add it back with ceph-volume lvm create --data /dev/sdX (or re-use a prepared device) and set the CRUSH weight so the fill rate stays inside your tolerance.
5. Throttle recovery if the cluster shares disks with production: ceph config set osd osd_max_backfills 1 and adjust osd_recovery_max_active rather than letting the default flood the network.

Two configuration items cause most avoidable Ceph incidents: osd_pool_default_min_size set too high, which makes a two-OSD-up cluster unable to write, and osd_pool_default_size higher than the failure domain can satisfy. Check both before any maintenance window.

Related reading: LVM thin provisioning, RAID levels and the parity write hole and Proxmox VE bridge and VLAN configuration.

原文链接：https://docs.ceph.com/en/squid/rados/troubleshooting/troubleshooting-pg