Challenge
Survive primary failover without lost writes or split-brain
An account service writes to a replicated database with async replication and unfenced failover.
The primary crashes at 4.5 ms: unshipped writes are lost (ackedWriteLost) and the old leader briefly accepts writes (splitBrainWrite). Fix both.
Goodput
≥ 94.0%
every seed
Budget
$500/mo
★★ at ≤ $400
Traffic
100 req/s (constant)
deadline 40 ms
Failure · db
kill
at t = 11 ms
- Client
- Load balancer
- Service
- Database
- Dedup store
- Cache
- ackedWriteLost
- splitBrainWrite