Rate limiter
Protect an API from overload and abuse by rejecting excess requests early with 429. The design choices are the algorithm, where the counters live, and what happens when the limiter itself is slow or down.
Component labels and connections
- Clients (client) · calls: API gateway
- API gateway (apiGateway) · calls: Rate limiter (shared counters)
- Rate limiter (shared counters) (rateLimiter) · calls: Load balancer
- Load balancer (loadBalancer) · calls: API 1, API 2
- API 1 (service) · calls: Orders DB
- API 2 (service) · calls: Orders DB
- Orders DB (database) · calls: none
Requirements
Functional
- Limit requests per client (API key, user or IP) per rule, e.g. 100 requests/minute, with short bursts allowed.
- Rejected requests get 429 Too Many Requests with Retry-After and rate-limit headers.
- Rules are configurable per endpoint and per plan without a deploy.
Non-functional
- Adds under 1 ms p99 to each request.
- Limits hold across all API servers (a client cannot multiply its quota by hitting different servers), within a small tolerance.
- A limiter failure must not take the API down: decide fail-open or fail-closed explicitly.
Worked estimates
Check rate
50k counter operations/s
every API request is checked: 50k req/s peak
Counter state
≈ 150 MB — fits in memory on one Redis node
1M active clients × 3 rules × ~50 B (key, count, timestamp)
Counter store throughput
1 shard suffices; run 3 shards for headroom and blast radius
~100k simple ops/s per Redis shard
Latency budget
≈ 0.6 ms per check, inside the 1 ms budget
same-AZ round trip ≈ 0.5 ms + Redis op ≈ 0.1 ms
Protected capacity
≈ 3,200 req/s; the global limit is set to 2,000/s with a 400-token burst, below saturation
2 API servers × 16 workers ÷ ~10 ms per request (simulated cell)
API
Any API call
Request: API key / user identity
Response: pass-through · 429 { error } + Retry-After, RateLimit-Limit, RateLimit-Remaining
PUT /internal/limits/{rule}
Request: { match, limit, window, burst }
Response: 200
Rules are pushed to limiters through config, cached locally.
Data model
- orders — store: db
Counter state is a key per (client, rule): token bucket = (tokens, lastRefill); sliding window counter = (current count, previous count, window start).
Keys carry a TTL slightly longer than the window, so idle clients cost nothing.
The orders entity is the protected backend's data, not limiter state.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Choosing the algorithm
Token bucket: a bucket of capacity B refills at r tokens/s; a request takes a token or is rejected. It allows bursts of B and a sustained rate of r, and needs two numbers per key. Leaky bucket smooths output to a constant rate (useful in front of a fragile dependency). Fixed window is one counter per window but lets 2× the limit through at a boundary. Sliding-window counter estimates the rolling count as current + previous × (overlap fraction), fixing most of the boundary problem for one extra number.
Where the counters live
Shared counters in Redis make limits global, at the cost of one round trip per request. Make the check atomic — a Lua script or INCR with EXPIRE — otherwise two servers read the same count and both allow. To cut the round trip, servers can batch: take tokens locally in chunks and sync every few hundred milliseconds, accepting a small overshoot. Shard counters by client key so one hot client cannot overload all shards.
Failure policy and layering
Decide per rule: abuse protection can fail open (an outage of the limiter should not be an outage of the API), while hard financial quotas may fail closed. Layer limits: a coarse per-IP limit at the edge or CDN against floods, per-key quotas at the gateway, and concurrency limits (bulkheads) inside services for overload that rate limits cannot see — a slow dependency at normal rate still exhausts workers.
Being a good 429
Return 429 with Retry-After so well-behaved clients back off, and the RateLimit headers so they can pace themselves. Clients must retry with exponential backoff and jitter; synchronised retries at exactly Retry-After create a new burst.
Failure scenarios
One client sends 6× the normal traffic
Scenario and expected result
From t = 1.3 s to 2.3 s the offered load jumps from 1,000 to 6,000 req/s — a misbehaving batch job or a scraper.
Expected: The limiter admits about 2,000 req/s plus the burst and answers the rest with 429 in well under a millisecond. Admitted requests stay fast, because the backend never runs past about two-thirds utilisation. Goodput falls, since a 429 is a failure from the client's point of view. The simulated limiter is one global bucket, so it rejects well-behaved clients too; with per-client buckets the 429s would land on the abusive client alone.
The counter store becomes slow
Scenario and expected result
From t = 1.5 s to 2.3 s every limiter check takes 50× longer — a Redis failover or a hot shard.
Expected: Every request pays the slowdown, because the limiter sits in the request path. Latency rises for all clients even though nobody is over quota. Production limiters put a tight timeout on the counter call and fail open (allow) when it expires.
The limiter goes down
Scenario and expected result
The limiter is killed from t = 1.5 s to 2.0 s.
Expected: Requests time out for the whole outage: in this topology the limiter fails closed, so a protection layer became a single point of failure. Fail-open with a local fallback limit keeps the API up and still bounds the damage.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
No limiter
- Nothing to operate, no extra hop. Under a 6× burst the backend still completes about as many requests as it can — roughly the same goodput — but every request now waits in a full queue: median latency rises about 10×, failures arrive as slow timeouts instead of instant 429s, and the work spent on requests that time out is wasted.
Reference
- Inspect the architecture above and run the same scenario.
Local in-process limiters
- No network hop and no shared dependency to fail. But each server only sees its own traffic, so the effective limit is (per-server limit × number of servers) and drifts as the fleet scales or the load balancer skews. Good as a coarse first line; pair it with shared counters for per-client quotas.
Reference
- Inspect the architecture above and run the same scenario.
Sliding window instead of token bucket
- A fixed window lets a client send 2× the limit across a window boundary. A sliding-window log is exact but stores every timestamp; the sliding-window counter (weighting the previous window's count) is a cheap approximation that Cloudflare reported to be within a fraction of a percent of exact. A token bucket expresses 'sustained rate plus burst' directly, which is usually what a quota means.
Clarifying questions
- What identifies a client: API key, user, IP?
- Is the limit a hard quota (billing) or overload protection?
- Are bursts allowed, and how large?
- Must the limit be exact across all servers, or is a few percent of overshoot fine?
- What should happen if the limiter is unavailable?
Common mistakes
- Read-then-write counter updates that race across servers.
- A fixed window without acknowledging the 2× boundary burst.
- Putting the limiter in the request path with no timeout and no failure policy.
- Per-server limits presented as global limits.
- Using rate limits as the only overload protection.
Self-assessment rubric
requirements / levels 1-4
- 1/4 'Limit requests'.
- 2/4 Per-client limits with a window.
- 3/4 Burst policy, accuracy tolerance and latency budget.
- 4/4 Explicit fail-open/closed policy and rule configurability.
estimates / levels 1-4
- 1/4 None.
- 2/4 Request rate.
- 3/4 Counter memory and store throughput.
- 4/4 Latency budget per check and shard count with headroom.
api / levels 1-4
- 1/4 None.
- 2/4 Returns an error when limited.
- 3/4 429 with Retry-After and limit headers.
- 4/4 Rule management API and propagation.
data model / levels 1-4
- 1/4 None.
- 2/4 A counter per client.
- 3/4 Algorithm-specific state with TTLs.
- 4/4 Atomic update strategy and key sharding.
high level / levels 1-4
- 1/4 Limiter in the service code only.
- 2/4 Shared counter store.
- 3/4 Limiter at the gateway, before expensive work.
- 4/4 Layered edge + gateway + concurrency limits.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Names one algorithm.
- 3/4 Compares algorithms including boundary bursts.
- 4/4 Handles races, batching and failure policy.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions metrics.
- 3/4 Names limiter latency and availability as risks.
- 4/4 Defines 429-rate and fail-open alerts and a rollout plan for new rules.