Skip to content

Load shedding

Why an overloaded server that accepts everything ends up serving no one, and how to reject early, bound queues by deadline and choose what to drop.

25 min7 steps

By the end you can
  • Explain why goodput collapses when a server queues work it cannot finish before clients give up
  • Compute how much load to shed and the deepest queue that can still meet a deadline
  • Choose what to shed and where, using priority, cheap early rejection, LIFO or CoDel-style queues and client-side throttling

Throughput is not goodput

  • 80% load: throughput80 %
  • 80% load: goodput80 %
  • 150% load: throughput100 %
  • 150% load: goodput3 %
Illustrative. The server stays 100% busy answering clients that already gave up (Amazon Builders' Library).

Offered load rises from 80% to 150% of a server's capacity. It has an unbounded FIFO queue and clients time out after 1 s. What happens to goodput after a while?

Answer this step to continue.

What this lesson simplifies

  • Overload is modelled with constant rates and a FIFO queue; real arrivals are bursty and service times vary.
  • Rejection is assumed to cost nothing; in practice it costs a little CPU and must stay far cheaper than serving.
  • Client behaviour after rejection, such as retries, is covered in the retries lesson and not modelled here.

Sources

  1. D. Yanacek, Amazon Builders' Library. Using load shedding to avoid overload.
  2. A. Forero Cuervo (B. Beyer et al., eds.). Handling Overload, Site Reliability Engineering, 2016.
  3. K. Nichols, V. Jacobson. Controlling Queue Delay, 2012.
  4. B. Maurer. Fail at Scale, 2015.

Reviewed by T2 content review on . Revision 1.