Throughput and Little's law
Relate arrival rate, concurrency and mean latency, estimate worker capacity, and recognize a queue that cannot reach steady state.
24 min8 steps
By the end you can
- Calculate mean in-flight requests from throughput and mean response time
- Estimate theoretical capacity from worker count and service time
- Identify overload and choose bounded admission rather than unbounded queues
Rate is different from time
Latency is how long one request takes. Throughput is completions per second. Label offered, completed, rejected and queued separately.
- Offered2,000 /s
- Completed1,200 /s
- Rejected300 /s
- Backlog growth500 /s
Use Little's law with matching units
L = λW for a stable system. Include queue wait in W if you include queued requests in L. It is accounting, not CPU sizing.
Throughput λ
2,000/s
Time W
50 ms
In flight L
100
2000 × 0.050
Loading question…
Draw the system boundary first
L and W must describe the same boundary. Use a long enough window; one snapshot is not the average.
Estimate worker capacity, then leave room
One worker at 25 ms/job: 40 jobs/s. Eight workers: at most 320 jobs/s, if nothing shared saturates. Leave headroom; there is no universal target.
Loading question…
A queue cannot manufacture capacity
Arrivals 500/s, capacity 400/s: backlog grows by 100 jobs every second. A bigger queue only delays the failure. Track queue age, not just length.
- 10 s1,000 jobs
- 60 s6,000 jobs
- 600 s60,000 jobs
Do not let the load test hide the queue
A fixed-concurrency client sends less when the service slows. An open, rate-driven test keeps offering load. Use both.
Closed loop
- Waits for replies
- Hides overload
Open arrival
- Constant rate
- Shows backlog and rejects
Scaling one tier can expose the next bottleneck
Stateless workers help only until a shared resource saturates.
Check your mastery, then observe a queue
Score 80% or more. Say whether time includes waiting and whether workers are independent.
Mastery check
5 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
In the sandbox, raise the arrival rate and watch completions flatten while waiting rises. Then read queues.
Answer this step to continue.