Proxmox VE Quorum Loss: Corosync Recovery Steps

When a Proxmox VE cluster loses quorum, the symptoms are dramatic and misleading: nodes flicker between green and grey in the GUI, configuration writes fail because /etc/pve goes read-only, and HA-managed guests get watchdog-reset. The dangerous part is the recovery reflex – pvecm expected 1 run carelessly can permanently fork a cluster. This article shows how to read the quorum state, how to tell network problems from node failures, and the order of operations that restores a cluster safely.

Read the state before touching anything

pvecm status
pvecm nodes
cat /etc/pve/corosync.conf
journalctl -u corosync -n 100 --no-pager | grep -Ei "TOTEM|quorum"

The output tells you three separate things and you need all three:

• Expected votes versus total votes – with 5 expected and 2 present, quorum needs 3 and Quorate: No.
• Membership – node flags: M member, x lost, d dead (not in cluster), N not a member.
• Transport and auth – Transport: knet and Secure auth: on are the normal values; anything else suggests a hand-edited corosync.conf.

[TOTEM] Retransmit List and A processor failed, forming new configuration entries mean packets are being lost or delayed on the cluster network, which is a network problem wearing a corosync costume – the majority of PVE cluster outages trace back to it.

Why the cluster behaves so badly

Corosync provides membership, quorum (via corosync_votequorum) and ordered messaging, and pmxcfs uses it to keep /etc/pve consistent. Without quorum, pmxcfs goes read-only, so GUI and API writes fail. Meanwhile the HA stack (pve-ha-crm and pve-ha-lrm) treats loss of quorum as a signal that the node may be isolated and triggers watchdog-based self-fencing – that is why guests appear to move or restart "by themselves" during a cluster incident.

The safe recovery order

1. Fix the network first. Verify the ring addresses in corosync.conf are reachable on the intended links and that no switch port or VLAN change is blocking multicast/unicast knet traffic.
2. Restart corosync where it is stuck – systemctl restart corosync (and pve-cluster if pmxcfs is degraded), then re-check pvecm status. Nodes typically rejoin and quorum returns automatically once they can talk.
3. Remove permanently dead nodes – if a node is gone for good, pvecm delnode pve-04 lowers the vote requirement, and the remaining nodes regain quorum.
4. Force only as a last resort – pvecm expected 1 tells the local node to ignore the vote count. It is legitimate when all nodes are alive but partitioned and you need a temporary single-node cluster to make changes; it is destructive when used as a habit, because it can permanently split the cluster.

systemctl status corosync pve-cluster
systemctl restart corosync
pvecm expected 1     # temporary, documented use only
pvecm status

Two-node clusters

A two-node cluster has no majority to fall back on: losing one node always means losing quorum unless a QDevice provides the third vote. The standard advice is to add an external QDevice on a small independent VM or host, keeping it reachable from both nodes and out of the same failure domain as the cluster network. Without it, every routine node reboot turns into manual quorum surgery.

Prevention checklist

• Use dedicated cluster links (or redundant links with a second ring0_addr) rather than sharing a busy uplink.
• Do not hand-edit totem timers: PVE ships correct defaults for LAN clusters, and tuning them usually creates the flapping you were trying to fix.
• Alert on Quorate transitions and on corosync retransmit messages, not just on node ping.
• Document the failback and rejoin procedure for a node that has been off-net for a long period.

Related: Proxmox VE VLAN-aware bridges and LACP bonds, Proxmox VE Linux bridge VLANs, and the storage side in Ceph OSD down triage.

原文链接：https://runbook.academy/courses/proxmox/lessons/xi-clustering-quorum-recovery