Disaster Recovery
Planning for the loss of a whole environment — what a DR plan must contain, and the drills that keep it true.
High availability handles the failure of a node. Disaster recovery handles the failure of everything in one place at once: a region outage, an account compromise, a deletion that replicated everywhere.
The distinction matters
What the plan must contain
A DR plan that works during an incident is specific and short:
- Declared scenarios. Region loss, data corruption, credential compromise, accidental deletion. Each has a different response.
- RPO and RTO per scenario, agreed with the business rather than assumed by the platform team.
- The restore procedure, written as commands, not prose. Include how to obtain credentials and keys when the primary environment is unavailable.
- Dependency order. Which systems must come back first for the database to be useful, and which must wait.
- Decision authority. Who declares a disaster and who authorises failover. Ambiguity here costs more time than any technical step.
- Communication. Where status is posted, who talks to customers.
Keep the plan somewhere reachable when the primary environment is down — a document stored only in the affected cloud account is not a plan.
Strategies and their cost
Cross-region asynchronous replication is the usual middle ground: acceptable RPO, an RTO measured in minutes, and no impact on primary write latency.
Drills
A plan that has never been executed is a hypothesis. Run a real drill at least annually, and after any significant architecture change:
- Restore into a fresh environment from backups alone, without touching the production cluster.
- Time each phase and compare against the stated RTO.
- Have someone who did not write the plan execute it. Every undocumented assumption surfaces immediately.
- Record what was missing — invariably a credential, a DNS record, a firewall rule or an extension.
Protecting against destructive changes
Region failure is rare; a bad statement is not. Two cheap defences:
- Delayed replica. A standby held an hour behind gives you an hour to notice a destructive statement and extract the pre-damage state without a full restore.
- Immutable backup retention. Object lock or write-once storage so that a compromised credential cannot delete the backups it can reach.