Disaster recovery: RPO, RTO and backups
Set recovery point and time objectives, see what really bounds them, why replicas are not backups, and how to choose and rehearse a DR strategy.
30 min7 steps
By the end you can
- Distinguish recovery point objective from recovery time objective for a given incident
- Compute worst-case data loss from the backup and log-shipping design
- Estimate restore time from data size and restore throughput, and list what else RTO includes
- Explain why replicas do not protect against deletion or corruption, and what makes a backup trustworthy
- Choose between backup-and-restore, pilot light, warm standby and active-active for an RPO and RTO
Two numbers: how much data, how long
RPO window
Committed writes you may lose
Incident
Region lost
RTO window
Time until service works again
Recovered
Serving
What sets the RPO
- Nightly snapshot86,400 s
- Snapshot + shipped WAL300 s
- Async replica (lag)5 s
- Synchronous replica0.1 s
Log scale: each gridline step is 10×.
Loading question…
Replicas are not backups
Replica
- DROP TABLE reaches all in seconds
- Ransomware too
Backup that works
- Isolated credentials
- Immutable
- Point-in-time
- Restored regularly
GitLab, 2017
- Wrong directory deleted
- No quick restore
- About 6 h of data lost
Restores take time you can calculate
Detect
Alert fires
Decide
Often the slowest human step
Provision
Capacity
Copy + replay
Size / throughput (bytes vs bits)
Verify
Integrity checks
Shift
DNS TTLs, connection pools
Loading question…
Rehearse the recovery
1
Stop the damage
2
Pick last good time
3
Restore snapshot, separate instance
4
Replay logs to just before
5
Verify
6
Reconcile, cut over
Put the steps of recovering from a destructive bad deploy in order.
- 1Verify the restored data with integrity checks and spot comparisons
- 2Replay archived logs up to just before the bad writes
- 3Stop the damage: roll back the deploy or block the writes that corrupt data
- 4Reconcile later good writes and shift traffic to the restored database
- 5Restore the most recent snapshot before that point into a separate instance
- 6Find the last good point in time from logs and the deploy timeline
Pick a DR strategy by RPO, RTO and cost
Backup and restore
- RPO hours, RTO hours
- Cheapest
Pilot light
- RPO minutes
- RTO tens of minutes
Warm standby
- RPO seconds-minutes
- RTO minutes
Active-active
- RPO and RTO near zero
- About double the cost
Check your mastery
Data loss
RPO
writes since last copy
Restore
size / throughput
bytes vs bits
Score 80% or more. Next: multi-region.
Mastery check
4 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
Answer this step to continue.