MongoDB Replica Sets

A replica set is a group of mongod processes holding the same data. One is the primary and accepts writes; the others replicate its oplog.

rs.initiate({
  _id: "shop-rs",
  members: [
    { _id: 0, host: "mongo-a.internal:27017", priority: 2 },
    { _id: 1, host: "mongo-b.internal:27017", priority: 1 },
    { _id: 2, host: "mongo-c.internal:27017", priority: 1 }
  ]
});
 
rs.status();
rs.conf();

The oplog

The oplog is a capped collection of idempotent operations. Its size determines how long a secondary can be offline and still catch up without a full resync.

db.getReplicationInfo();     // oplog size and the time window it covers
rs.printReplicationInfo();

Elections

When members stop receiving heartbeats from the primary within electionTimeoutMillis (10 seconds by default), the remaining members hold an election. A candidate needs votes from a majority of all voting members, and only a member whose oplog is at least as current as the others can win.

The consequences:

• A replica set of three tolerates one member loss; five tolerates two.
• With only two members reachable out of three, one can still be elected. With one out of three, no primary exists and the set is read-only.
• Deploy across three failure domains. Two zones is the worst arrangement: losing the zone with two members leaves no majority.

Election-related settings:

cfg = rs.conf();
cfg.members[0].priority = 2;        // preferred primary
cfg.members[2].priority = 0;        // never becomes primary
cfg.members[2].hidden = true;       // invisible to drivers; for backups or analytics
cfg.members[2].secondaryDelaySecs = 3600;   // delayed member, protects against bad writes
cfg.settings.electionTimeoutMillis = 10000;
rs.reconfig(cfg);

Arbiters

An arbiter votes but holds no data. It makes a two-member set able to elect a primary — and it is usually the wrong answer.

Rollback

If a primary accepts writes that were not replicated to a majority and then fails, those writes are rolled back when it rejoins. MongoDB writes the rolled-back documents to a file rather than discarding them silently, but reconciling them is manual.

Using w: majority prevents an acknowledged write from being rolled back — the reason it is the right default. See Write Concerns.

Monitoring

rs.status().members.forEach(m => print(
  m.name, m.stateStr, m.health,
  "lag:", (m.optimeDate ? (new Date() - m.optimeDate) / 1000 : "n/a") + "s"
));

Alert on: no primary present, a member not in PRIMARY or SECONDARY state, replication lag beyond your staleness budget, and the oplog window falling below your maintenance requirement.