MongoDB Replica Set Configuration and Failover Operations

A MongoDB replica set gives you automatic failover and read scaling, but only if the members agree on what a quorum is and which node should become primary. The failure modes are well defined: a member whose oplog has wrapped must be resynced from scratch, an arbiter cannot hold data, and an election priority set incorrectly can hand the primary role to a node in the wrong data centre. This guide covers initiating a replica set properly, adding members, tuning election behaviour, and recovering a member that has fallen off the oplog.

Replica Set Fundamentals

Voting members and quorum

Every member holds a complete copy of the data except arbiters, which exist only to break ties in elections. A set needs an odd number of voting members for a clear majority: three data-bearing members is the minimum sensible production topology, and adding a fourth data-bearing member does not improve fault tolerance but does add a vote, which is why the fourth is usually a hidden or priority-0 member.

Members communicate over the oplog — a capped collection of operations — and a secondary applies operations in order. If a secondary is offline long enough that the operations it needs have been overwritten in the primary's oplog, it cannot catch up incrementally and must be resynchronised.

Step 1 - Configure Each Member

Every member should be started with the same replica set name and a distinct bind address. The replica set name must match exactly or the member will refuse to join:

mongod --replSet rs0 --port 27017 --dbpath /var/lib/mongodb --bind_ip 10.10.30.11 --oplogSize 10240

Always set oplogSize explicitly. The default sized oplog is derived from free disk space, which means a large data volume gets a large oplog and a small one gets an oplog so short that a brief maintenance window causes a full resync.

Step 2 - Initiate the Replica Set

Connect to the first member and initiate with an explicit configuration document rather than accepting the single-member default:

mongosh --host 10.10.30.11

rs.initiate({
  _id: "rs0",
  members: [
    { _id: 0, host: "10.10.30.11:27017", priority: 2 },
    { _id: 1, host: "10.10.30.12:27017", priority: 1 },
    { _id: 2, host: "10.10.30.13:27017", priority: 1 }
  ]
})

Use resolvable hostnames rather than IPs where your DNS is reliable — a replica set stores member addresses in the configuration, and changing them later requires a reconfiguration that briefly affects the set. The priority field drives elections: the highest-priority member that is up to date becomes primary.

Step 3 - Add Members and Tune Elections

Add a member and adjust election behaviour with rs.add and rs.reconfig:

rs.add({ host: "10.10.30.14:27017", priority: 0, hidden: true })
rs.status()
rs.conf()

A member with priority 0 can never become primary — useful for a reporting replica. A hidden member is invisible to client reads and is the standard way to run a backup or analytics copy. Do not make a member in a remote data centre the highest priority unless you intend it to become primary; that configuration is how cross-site writes pick up tens of milliseconds of latency.

To change an existing member's priority:

cfg = rs.conf()
cfg.members[2].priority = 0
cfg.members[2].hidden = true
rs.reconfig(cfg)

Step 4 - Verify Replication Health

rs.status()
rs.printReplicationInfo()
rs.printSecondaryReplicationInfo()
db.serverStatus().oplog
db.getSiblingDB("local").oplog.rs.stats().maxSize

Reading the oplog window

rs.printReplicationInfo() reports the oplog window — how much wall-clock time the oplog holds. A window of two hours is comfortable; a window of twenty minutes means any maintenance longer than that will force a full resync. That number is the single most useful replication health metric, and it should be monitored and alerted on.

rs.printSecondaryReplicationInfo() shows per-member lag in seconds. Sustained lag with a healthy oplog window usually points at disk throughput or an under-resourced secondary rather than a network problem.

Recovering a Member That Fell Off the Oplog

Two ways to resynchronise a member

When a member's oplog has wrapped, it reports a state of RECOVERING and cannot catch up. There are two approaches, and the right one depends on how much downtime you can accept.

Preferred: seed from an existing member. Stop the stale member, copy the data files from a healthy secondary, start it with a new member _id, and let it apply operations from the oplog until it reflects the current state:

# On the stale member: stop mongod, then move the old data aside
sudo systemctl stop mongod
sudo mv /var/lib/mongodb /var/lib/mongodb.old

# Copy from a healthy secondary (filesystem snapshot or rsync of the dbpath)
rsync -a --delete 10.10.30.12:/var/lib/mongodb/ /var/lib/mongodb/
sudo chown -R mongodb:mongodb /var/lib/mongodb
sudo systemctl start mongod

This is much faster than a full initial sync, because the copied data is already a recent snapshot and only the operations since the snapshot need to be applied.

Alternative: let it initial sync. Delete the data directory and restart the member; it will pull a complete copy from the primary. Simpler, but it transfers the entire dataset and puts sustained read load on the primary, which is a real risk in a busy cluster.

Either way, confirm recovery with rs.status() — the member should move from STARTUP2 through RECOVERING to SECONDARY, and its lag in rs.printSecondaryReplicationInfo() should reach zero.

Operational Notes

• Set oplogSize explicitly; the default is derived from disk space and is often too small.
• Monitor the oplog window, not just lag — window is the number that predicts whether a restart will need a resync.
• Keep the voting membership odd and keep the number of voting members low.
• Use priority 0 and hidden for reporting or backup members so they never take the primary role.
• Prefer restoring from a healthy member's data files over a full initial sync; the primary will thank you.

Related reading: our MySQL 8 GTID replication setup and troubleshooting guide, the PostgreSQL 16 streaming replication guide, and the Redis persistence RDB versus AOF article.

原文链接：https://www.mongodb.com/docs/manual/tutorial/resync-replica-set-member/