Circuit breakers and bulkheads
Stop calling a dependency that is failing, and stop one slow dependency from taking every thread: breaker states, trip thresholds and bulkhead sizing.
30 min8 steps
By the end you can
- Trace a circuit breaker through closed, open and half-open, and say what each state protects
- Calculate when a failure-rate breaker trips and choose its window, minimum calls and scope
- Size a bulkhead with Little's law and predict the throughput it admits when the dependency hangs
Failing fast beats failing slowly
No breaker
- Each call waits a full timeout
- Threads pile up
- Caller dies too
Breaker open
- Calls fail in microseconds
- Threads stay free
- Dependency gets quiet to recover
Closed, open, half-open
Closed
Calls flow, failures counted
Trip
Failure rate over threshold
Open
All calls rejected during cool-down
Half-open
A few probe calls pass
Closed
Probes succeed
A dependency fails for a minute, then recovers. Put the breaker's behaviour in order.
- 1Open: calls are rejected immediately during the cool-down
- 2Half-open: a few probe calls are let through
- 3Closed: calls pass and failures start to accumulate
- 4The probes succeed and the breaker closes again
- 5The failure rate crosses the threshold and the breaker opens
When does it trip?
Window
last N calls
or last T seconds
Failure rate
e.g. 50%
not a raw count
Min calls
e.g. 20
no trip on 2 of 3 at 3 a.m.
Slow calls
count as failures
catches slow decay
Loading question…
Scope: one breaker for what?
Bulkheads: compartments for failure
Loading question…
What the bulkhead buys you when things hang
Stuck calls
20 max
excess fail fast
Other endpoints
unaffected
Admitted rate
slots / latency
20 / 2 s = 10 per s
Loading question…
Breakers, bulkheads, timeouts and retries together
Timeout
- One call's duration
Bulkhead
- Concurrent calls
Breaker
- Calls after evidence of failure
Retry
- Recovers transient errors
- Counts as extra calls
Check your mastery, then trip a breaker
Breaker
opens
downstream load drops
Bulkhead
slots / latency
compare rejections
Score 80% or more. In the sandbox, build client, service, circuit breaker, downstream. Make it return 503, then slow it with a bulkhead set.
Mastery check
5 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
Answer this step to continue.