DRBD 9 Setup: Resources, Promotion and Quorum

DRBD provides synchronous block-level replication between two Linux hosts, which is what makes a two-node cluster with a shared-looking disk possible without shared storage hardware. Version 9 adds multi-node support and, more importantly for operations, quorum - the mechanism that stops a partitioned node from silently becoming a second primary. This guide covers the resource definition, first-time bring-up, and the recovery procedure when a split-brain has already happened.

The resource file

# /etc/drbd.d/r0.res
resource r0 {
  options {
    quorum majority;
    on-no-quorum io-error;
  }
  device /dev/drbd0;
  meta-disk internal;

  on node1 {
    node-id 0;
    address 10.10.10.11:7789;
    disk /dev/vg0/data;
  }
  on node2 {
    node-id 1;
    address 10.10.10.12:7789;
    disk /dev/vg0/data;
  }
  net {
    protocol C;      # synchronous replication: write acked after both nodes
  }
}

Protocol C is the usual choice for high availability: a write is acknowledged only after the peer has written it, so failover does not lose acknowledged data. Protocol A/B trade durability for latency. meta-disk internal stores metadata at the end of the backing device, so the backing device must be large enough to hold both data and metadata.

Bring-up on both nodes

# drbdadm create-md r0
# drbdadm up r0
# drbdadm status r0
r0 role:Secondary
  disk:Inconsistent
  node2 role:Secondary
    peer-disk:Inconsistent

# one node only, first time: declare which side wins
# drbdadm primary --force r0

Initial synchronisation can take hours on multi-terabyte volumes. If the backing devices are genuinely empty on both sides - a freshly provisioned pair - skip the full sync deliberately with drbdadm new-current-uuid --clear-bitmap r0 before promoting, rather than waiting for a copy of zeros.

Promotion, demotion and status

# drbdadm primary r0
# drbdadm secondary r0
# drbdadm status
# drbdadm status r0 --verbose
# cat /proc/drbd -- obsolete in DRBD 9; use drbdadm status instead

In DRBD 9 drbdadm status (or drbdsetup status) replaces the old /proc/drbd output. Read it for three things: local role (Primary/Secondary), local disk state (UpToDate / Inconsistent / Diskless), and the per-peer connection state (Connected, WFConnection, StandAlone). StandAlone almost always means the replication link failed or an administrator disconnected the resource.

Quorum: the anti-split-brain control

options {
  quorum majority;
  quorum-minimum-redundancy 2;
  on-no-quorum suspend-io;        # alternative to io-error
}

Quorum is evaluated across the nodes that are actually configured and connected. quorum majority requires more than half of the visible nodes; on-no-quorum io-error makes writes fail fast so the application and monitoring see the problem, while suspend-io freezes I/O and resumes automatically when quorum returns. For a two-node cluster, quorum arithmetic alone cannot prevent a partition disagreement, which is why a third lightweight tie-breaker node or an out-of-band fencing mechanism matters.

Recovering from a split-brain

The symptom is two nodes both reporting Primary with StandAlone connections, and diverged data. Decide which side holds the authoritative data, then on the losing node:

# drbdadm disconnect r0
# drbdadm secondary r0
# drbdadm connect --discard-my-data r0
# on the surviving node:
# drbdadm connect r0

After the connection re-establishes, DRBD resynchronises the discarded side. Discarding data is irreversible - if there is any doubt about which side is correct, stop the application, copy the newer data off manually, and only then resynchronise.

Operations that avoid the incident

• Monitor drbdadm status from both nodes and alert on StandAlone, Inconsistent or Diskless - not only on node-down events.
• Keep the replication link on its own path; a busy management network is a common cause of connection loss.
• Never promote both nodes manually "to be safe"; that is how split-brain is manufactured.
• After growing the backing device, run drbdadm resize on the primary and one on each peer before resizing the filesystem.
• Test the failover path: promote the secondary during a maintenance window and confirm the application starts against /dev/drbd0.

Related: Linux LIO iSCSI target configuration, Linux multipath for iSCSI SAN, and Proxmox VE quorum loss recovery.

原文链接：https://linbit.com/drbd-user-guide/drbd-guide-9_0-en/