Disaster Recovery Plans
Writing a DR plan that works during an incident, and setting recovery objectives that are measured rather than asserted.
A disaster recovery plan is a document someone follows under pressure, at an unfamiliar hour, with part of the infrastructure missing. Write it for that reader.
Recovery Point Objective
The RPO is the maximum acceptable data loss, expressed in time. It determines the backup and replication design directly:
Recovery Time Objective
The RTO is the maximum acceptable downtime. It must be measured, not asserted.
Stated RTO: 1 hour
Measured restore: 3 hours 40 minutes for 2 TB from object storage
Conclusion: backups alone cannot meet this. A standby is required.Include every phase, not just the data copy: detecting the failure, deciding to declare a disaster, provisioning infrastructure, restoring, verifying, redirecting traffic and confirming the application works. The decision step is frequently the longest one.
What the plan contains
- Scenarios, each with its own response: region loss, data corruption, accidental deletion, credential compromise, provider outage.
- RPO and RTO per scenario, agreed with the service owner in writing.
- The procedure as commands, not prose. Copy-pasteable, with the actual host names and paths.
- How to obtain credentials and keys when the primary environment is unavailable. This is the step most plans omit and most drills fail on.
- Dependency order — which systems must come back before the database is useful.
- Decision authority — who declares a disaster, who authorises a failover.
- Communication — where status is posted, who informs customers.
- The fail-back procedure, for returning to the primary environment afterwards.
Keeping it true
A plan decays as infrastructure changes. Two habits keep it accurate:
- Review it after every significant architecture change — a new region, a new engine version, a changed backup tool.
- Execute it on a schedule, with someone who did not write it. See Restore Testing.
Recording the outcome
After each drill, record: the measured RTO per phase, what was missing, and what was changed as a result. A plan with a history of drills and corrections is trustworthy; one that has never been executed is a document expressing intent.