Notification system
Accept notification requests from many internal services and deliver them through push, SMS and email providers. The design is a durable queue between fast producers and slow, flaky third parties, with retries, rate limits and user preferences in between.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Notify API 1, Notify API 2
- Notify API 1 (service) · calls: Idempotency keys, Notification queue
- Notify API 2 (service) · calls: Idempotency keys, Notification queue
- Idempotency keys (dedupStore) · calls: none
- Notification queue (queue) · calls: none
- Sender workers (worker) · calls: Breaker (per provider)
- Breaker (per provider) (circuitBreaker) · calls: Push provider (APNs/FCM)
- Push provider (APNs/FCM) (service) · calls: none
Requirements
Functional
- Internal services send a notification for a user and a template; the system picks channels (push, SMS, email) from the user's preferences.
- Respect opt-outs, quiet hours and per-user frequency caps.
- Track status (queued, sent, delivered, failed) and retry transient failures.
Non-functional
- Accepting a request takes under 50 ms and never depends on provider health.
- At-least-once delivery to the provider, with duplicates suppressed where the channel allows (collapse IDs, idempotency keys).
- Time-sensitive notifications (2FA codes) delivered within seconds; marketing can wait.
Worked estimates
Volume
≈ 11.6k/s average; campaigns spike to 10× for minutes
1B notifications/day ÷ 86,400
Queue backlog during a provider outage
≈ 7M messages ≈ 7 GB at 1 KB — a durable log handles it; an in-memory queue does not
11.6k/s × 10 min
Sender concurrency
≈ 460 requests in flight; HTTP/2 multiplexing keeps that to a few hundred connections
Little's law: 11.6k/s × 40 ms provider latency
Queue partitions
≥ 460 partitions, or unordered consumption; the simulated cell needs 1,000/s × ~45 ms ≈ 45 and has 128
a partition is consumed one message at a time, in order → partitions ≥ in-flight messages
Status records
≈ 6 TB — a wide-column or time-partitioned store
1B/day × 200 B × 30 days retention
API
POST /v1/notifications
Request: { userId, template, params, priority } + Idempotency-Key
Response: 201 { notificationId } queued · 200 duplicate · 429
GET /v1/notifications/{id}
Request: —
Response: 200 { status, attempts }
PUT /v1/users/{id}/preferences
Request: { channels, quietHours, optOuts }
Response: 200
Data model
notifications(notificationId, userId, template, channel, status, attempts, createdAt): written on accept, updated by senders.
preferences(userId → channels, opt-outs, quiet hours): read on every send, so it is cached aggressively.
devices(userId → push tokens): tokens rotate; providers report invalid tokens, which must be deleted.
The idempotency store maps the caller's key to a notificationId for a day.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Delivery semantics end to end
The API records an idempotency key and enqueues; the caller's retry is a no-op. Senders acknowledge a message only after the provider accepts it, so a crash means redelivery: at-least-once. Duplicates reaching the user are suppressed per channel — APNs collapse IDs and FCM collapse keys replace an undelivered earlier notification, email providers accept idempotency keys. After N attempts a message goes to a dead-letter queue for inspection instead of retrying forever.
Retries, breakers and provider limits
Retry transient errors (timeouts, 5xx, 429) with exponential backoff and jitter; never retry permanent ones (invalid token — delete it). Providers enforce rate limits, so senders pace themselves with a token bucket per provider and per app. A circuit breaker per provider stops hammering a provider that is failing, and a bulkhead caps concurrent calls so one slow provider cannot starve the others.
Priorities, preferences and campaigns
A marketing campaign of 50M notifications must not delay a 2FA code: separate queues by priority, or at least separate consumer pools. Preferences and frequency caps are evaluated at send time (not enqueue time), because users change them and quiet hours depend on when delivery actually happens. Campaigns are rate-shaped at the source so they do not hit the queue as a single burst.
Failure scenarios
The push provider fails half of all requests
Scenario and expected result
From t = 1.3 s to 1.9 s the provider returns errors for 50% of requests while still answering health checks.
Expected: The API keeps accepting at full speed — it only writes to the queue. Senders see failures, retry once, then leave messages for redelivery; the breaker opens and fails fast instead of piling load onto the provider. After the incident the redelivered messages go through. Watch the queue's redelivery count rather than the API's goodput.
The provider is down long enough to fill the queue
Scenario and expected result
The provider is unreachable from t = 1.1 s to the end of the run. The queue holds 1,000 messages — about one second of traffic.
Expected: For a while nothing changes for callers. Then the queue reaches capacity and publishes are rejected with 503: back-pressure reaches the API. That is the right failure — loud and bounded — but the capacity must cover the longest provider outage you plan for, which is why production systems use a disk-backed log sized in hours.
The provider's latency jumps by 250 ms
Scenario and expected result
From t = 1.3 s to 2.3 s every call from the breaker to the provider takes 250 ms longer.
Expected: Each sender now waits ~290 ms per notification, so 128 senders deliver only ~440/s against 1,000/s arriving (Little's law). The queue backs up, and well before the slowdown ends it reaches its 1,000-message capacity: new requests get 503. Latency alone produced the same back-pressure as an outage. The bulkhead keeps the slowdown from consuming unbounded resources; autoscaling senders on queue depth and a deeper, disk-backed queue are the fixes.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
Call the provider synchronously from the API
- One less moving part and immediate status. But the caller's latency and availability now equal the provider's: when APNs slows down, every internal service that sends notifications slows down with it. The queue is what decouples them.
Reference
- Inspect the architecture above and run the same scenario.
Separate queues per channel and priority
- One queue per (channel, priority) lets 2FA codes bypass a marketing campaign backlog and lets each provider's senders scale and rate-limit independently. More queues to operate, but a slow email provider can no longer delay push.
Clarifying questions
- Which channels, and which providers?
- Which notifications are time-critical?
- Is delivery confirmation required, or is 'handed to the provider' enough?
- Expected campaign sizes and how fast they must go out?
Common mistakes
- Calling providers synchronously in the request path.
- Retrying without backoff or a cap, so an outage becomes a retry storm.
- One queue for all priorities.
- Ignoring provider rate limits and invalid-token feedback.
- No idempotency key, so caller retries send duplicates.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Send notifications.
- 2/4 Multiple channels.
- 3/4 Preferences, priorities and delivery guarantees.
- 4/4 Separates acceptance latency from delivery latency and defines duplicate handling per channel.
estimates / levels 1-4
- 1/4 None.
- 2/4 Throughput.
- 3/4 Throughput, backlog during an outage, status storage.
- 4/4 Sender concurrency from Little's law and provider limits.
api / levels 1-4
- 1/4 Send endpoint.
- 2/4 Adds status.
- 3/4 Idempotency key and priority.
- 4/4 Preferences API and provider callbacks for delivery receipts.
data model / levels 1-4
- 1/4 None.
- 2/4 Notifications table.
- 3/4 Notifications, preferences, devices.
- 4/4 Token lifecycle and status retention.
high level / levels 1-4
- 1/4 API calls provider.
- 2/4 Queue between API and senders.
- 3/4 Per-channel senders with retries.
- 4/4 Priorities, breakers, bulkheads and DLQ with clear failure behaviour.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions retries.
- 3/4 At-least-once with idempotency and DLQ.
- 4/4 Provider rate limits, collapse keys and preference evaluation timing.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions metrics.
- 3/4 Queue depth and provider errors as signals.
- 4/4 Per-priority delivery SLOs with alerts on lag and DLQ growth.