Queues and delivery guarantees
Predict backlog and recovery time, trace acknowledgements through failures, and design safe redelivery, ordering and dead-letter handling.
25 min8 steps
By the end you can
- Calculate backlog growth and drain time with continuing arrivals
- Identify loss and duplicate windows around processing and acknowledgement
- Choose partition keys and concurrency for per-entity ordering
- Choose bounded retries, dead-letter handling and back-pressure
Accepted is not processed
A publish confirm means only what the broker's durability policy promises. It is not processing, and not atomic with your database commit. Use an outbox plus idempotent consumers.
A burst spends queue space
Backlog = (arrivals − completions) × time. A queue buys time in a finite burst. Track bytes and age, not just count.
Net growth
200/s
Backlog
6,000 jobs
(600 − 400) × 30
Loading question…
Spare capacity pays back the backlog
Drain at the spare capacity, with arrivals still coming. Dividing by the full 400/s is wrong.
Burst ends
6,000 queued; arrivals drop to 300/s
+30 s
3,000 left (100/s spare)
+60 s
Queue empty
Loading question…
The acknowledgement position chooses a failure window
Where you acknowledge picks the failure window. Ack-after-processing is at-least-once: duplicates are possible. Never assume a worker owns a message forever.
Ack before work
- Crash loses the job
Ack after work
- Crash means redelivery
- Effect must be idempotent
Order a consumer path that makes a database effect safe on redelivery.
- 1Receive the message and its stable business operation ID
- 2Commit the database transaction
- 3Acknowledge successful processing to the broker
- 4In one database transaction, record the ID and apply the effect only if it is new
Exactly once needs an explicit boundary
Idempotent consumer + at-least-once delivery gives an effectively-once effect, if the ID check and the mutation commit atomically. External APIs need their own idempotency key.
Ordering is usually per partition
Order holds per partition, not globally. Key by account; keep per-key sequential processing through retries. A hot key caps parallelism.
A dead-letter queue is a workflow, not a bin
Retry transient errors with bounded delay. After N attempts, park the message in a DLQ: alert, keep the cause, fix, replay with the same ID. Watch oldest-message age and DLQ growth.
Check your mastery, then crash a consumer
Score 80% or more. Say which rate includes retries and whether arrivals continue.
Mastery check
5 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
In the sandbox, stop the consumer while arrivals continue, then restore it and measure recovery. The model assumes constant rates.
Answer this step to continue.