Proxmox Corosync Link Priority and Token Tuning

A Proxmox cluster that reboots healthy nodes during a backup window is not failing because of a bug. Corosync's heartbeat is a small UDP packet that must arrive within the token timeout, and on a shared gigabit link a bulk transfer can delay it past that threshold. When two nodes cross the threshold simultaneously, quorum fractures and the watchdog fences the minority side. This article explains the latency-versus-bandwidth distinction that makes this failure mode so counterintuitive, and the configuration that prevents it.

Corosync cares about latency, not bandwidth

Heartbeat packets are tens to a couple of hundred bytes. A gigabit link carries them using a rounding error of its capacity. The problem is queueing: on an undifferentiated link the transmit queue is first in, first out, so a corosync packet sits behind whatever bulk traffic is currently being sent. Under saturation, that queueing delay can reach hundreds of milliseconds to seconds — comfortably past a 1000 ms token timeout with four retransmits.

A backup job pushing 850 Mbps across the same link that carries corosync is a textbook cause. Nothing failed; a latency-sensitive protocol simply lost a priority contest it should never have been in.

Fix the topology first

• Tier 1 — a dedicated physical NIC for corosync. Cheapest in engineering effort and most robust in outcome. An unmanaged five-port switch, or direct cables between nodes in a small cluster, removes contention entirely.
• Tier 2 — a dedicated VLAN with enforced priority. A VLAN alone does not fix queueing because it is the same wire and the same transmit buffer. You need the switch to actually honour a strict-priority or weighted queue for that traffic class.
• Tier 3 — knet link0/link1 redundancy. This solves hardware failure of the path, not congestion, and complements rather than replaces the tiers above.

Configuration

# Create a cluster with explicit link priorities
pvecm create prod-cluster \
  --link0 10.20.20.11,priority=20 \
  --link1 10.10.10.11,priority=10

# /etc/pve/corosync.conf
totem {
  version: 2
  cluster_name: prod-cluster
  config_version: 9
  transport: knet
  interface { linknumber: 0; knet_link_priority: 20 }
  interface { linknumber: 1; knet_link_priority: 10 }
  ip_version: ipv4-6
  link_mode: passive
  secauth: on
}
nodelist {
  node { name: pve1 nodeid: 1 quorum_votes: 1 ring0_addr: 10.20.20.11 ring1_addr: 10.10.10.11 }
  node { name: pve2 nodeid: 2 quorum_votes: 1 ring0_addr: 10.20.20.12 ring1_addr: 10.10.10.12 }
  node { name: pve3 nodeid: 3 quorum_votes: 1 ring0_addr: 10.20.20.13 ring1_addr: 10.10.10.13 }
}
quorum { provider: corosync_votequorum }

On Proxmox VE 9 the interface block carries only linknumber; addresses live per node as ring0_addr and ring1_addr. There is no bindnetaddr, no multicast address and no explicit transport line needed — knet is the default since corosync 3. link_mode: passive means only the highest-priority working link carries traffic, which is right for almost every cluster.

Apply without restarting the cluster

# after editing /etc/pve/corosync.conf: increase config_version, then
corosync-cfgtool -R
pvecm status | grep -E 'Config Version|Quorate'
corosync-cfgtool -s

What raising the token timeout actually costs

Every genuine node failure is detected later by exactly as much as you raised the timeout. Moving token to 3000 ms with 10 retransmits buys roughly 30 seconds of silence tolerance, and costs 30 seconds of extra downtime for a node that has truly died, on top of the approximately two minutes the recovery state machine already takes. Treat timeout tuning as a stopgap for topology work you have not finished, not as a substitute, and write the reason into your runbook so a future on-call engineer does not mistake it for a considered default.

# Only if you cannot isolate the link
totem {
  token: 3000
  token_retransmits_before_loss_const: 10
  token_coefficient: 125
}

Verify, then test deliberately

corosync-cfgtool -s
corosync-cfgtool -n
journalctl -b -u corosync | grep -E "Token has not been received|best link|is down"

The best link line is the one to watch: it names the link knet is currently preferring, which proves your priority ordering took effect rather than merely being written down. Then test failover on purpose — disable a switch port or bring the corosync interface down on one node during a maintenance window and confirm quorum holds. Do not do that test on a two-node cluster without a QDevice, where losing one link means losing quorum outright.

Finally, keep corosync off any WAN link. It is a LAN latency protocol, assumes sub-10 ms round trips, and misbehaves well before WAN latency figures are reached. If you are debugging cluster membership loss after a hardware change, the recovery procedure in Proxmox VE quorum loss and corosync recovery is the companion reference.

For the bridge and bond layout on the same nodes, see Proxmox VE VLAN-aware bridge with LACP bond.

原文链接：https://runbook.academy/courses/proxmox/lessons/xi-clustering-corosync-links-tuning