Skip to content

Failure modes and metastability

How cascades, retry storms, thundering herds, metastable states, split brain and gray failures turn a small fault into an outage, and how to break each loop.

35 min9 steps

By the end you can
  • Trace how a cascading failure spreads load from failed replicas to healthy ones, and name what stops it
  • Predict how retries at stacked layers multiply load on a failing dependency
  • Explain a metastable failure as a trigger plus a sustaining loop, and choose actions that recover from it
  • Estimate a thundering herd and spread it with jitter
  • Recognise split brain and gray failure, and the mechanisms that contain them

A cascade: the survivors inherit the load

Load balancerReplica 1107%Replica 2107%Replica 3107%Replica 4crashed
80% x 4 / 3 = 107%. Each replica must protect itself: cap work, shed cheaply, keep health checks independent of load.

During the cascade above, which change most directly stops it from spreading?

Answer this step to continue.