Skip to content
Navigation

Type at least two characters. Search covers page titles, headings, tags and database names.

↑ ↓ to navigateEnter to openEsc to close0 pages
Operationsadvanced

Failover Procedures

Planned switchover and unplanned failover, the fencing that prevents split brain, and what to verify afterwards.

3 min readAdvancedUpdated Edit this page

Failover has two forms, and conflating them causes incidents. A switchover is planned: the primary is healthy and hands over cleanly, with no data loss. A failover is unplanned: the primary is gone, and whatever it had not replicated is lost.

Planned switchover

  1. Verify the target replica is healthy and caught up — lag at or near zero.
  2. Stop or drain application writes.
  3. Confirm lag is zero.
  4. Demote the primary so it stops accepting writes.
  5. Promote the replica.
  6. Repoint remaining replicas at the new primary.
  7. Redirect clients.
  8. Verify writes succeed and replication is healthy in the new topology.

Steps 2 and 3 are what make a switchover lossless. Skipping them turns it into a failover with extra steps.

Unplanned failover

  1. Confirm the primary is actually gone, from more than one observer. A single monitoring failure is not evidence.
  2. Fence it. Ensure it cannot accept writes if it returns.
  3. Choose the most advanced replica. With several, promoting the one with the most data minimises loss.
  4. Promote.
  5. Redirect clients.
  6. Rebuild the topology: the new primary needs its own replica, its own backups and its own WAL or binlog archiving.
  7. Record what was lost — the replication position gap — and reconcile it deliberately.

Client redirection

This step causes more failed failovers than promotion does. Options, from most to least reliable:

  • A proxy (PgBouncer, HAProxy, ProxySQL, MySQL Router) that follows the cluster manager.
  • Driver-level discovery — MongoDB replica sets, Redis Sentinel-aware clients.
  • A virtual IP moved by the cluster manager.
  • DNS with a short TTL — simple, and only as fast as the slowest resolver cache in your stack.

Whichever you use, test that connection pools actually re-resolve. A pool holding a cached address keeps trying the dead node long after the failover completed.

After any failover

Checklist for the hour after:

  • New primary has at least one healthy replica.
  • Archiving and backups are running and verified against the new primary.
  • Monitoring and alerting target the new topology.
  • The old primary is either rebuilt as a replica or decommissioned — not left running ambiguously.
  • The data loss window, if any, is quantified and communicated.

Rehearse it