High Availability

High availability means the system keeps serving when a component fails. It is not a product you enable; it is a set of decisions about detection, promotion, redirection and fencing.

Translate the target into a budget

Include planned maintenance. If a minor-version upgrade takes ten minutes and you do it monthly, you have already spent most of a 99.99% budget on planned work — which is the argument for rolling upgrades rather than for a better failover script.

The four parts of a failover path

1. Detection. A quorum of observers must agree the primary is gone. A single watcher cannot distinguish a dead node from an unreachable one, and acting on one observer's opinion is how split brain starts.
2. Promotion. A replica becomes primary. With asynchronous replication, un-shipped writes are lost here.
3. Redirection. Clients must reach the new primary — via a proxy, a virtual IP, a DNS record with a short TTL, or driver-level discovery. This step, not promotion, is where most failovers actually stall.
4. Fencing. The old primary must be prevented from accepting writes if it returns.

Failure modes of HA systems

Adding automated failover adds failure modes of its own:

• Flapping. Aggressive health checks promote on a transient network blip, then again on the way back. Require several consecutive failures, and rate-limit promotions.
• The cluster manager becomes the single point of failure. If it needs a consensus store, that store needs its own quorum, its own monitoring and its own disaster plan.
• Failing over into insufficient capacity. A promoted replica sized for read traffic may not handle the write load. Size replicas as candidate primaries.
• The replica was never healthy. A replica that has been lagging for a week is not a failover target. Alert on lag and on the absence of a viable candidate, not only on node death.

Client behaviour during failover

The database being available again does not mean the application recovers. Ensure:

• Connection pools detect broken connections and re-resolve the endpoint rather than reusing a cached address.
• Retries are bounded and jittered, so the new primary is not hit by a synchronised retry storm the moment it accepts connections.
• Writes are idempotent where a retry after an ambiguous failure could duplicate them.