Chat (WhatsApp / Messenger)
One-to-one and group messaging with delivery receipts. The core problems are keeping millions of connections open, storing messages once even when clients retry, and delivering them in order to devices that come and go.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Connection gateway 1, Connection gateway 2
- Connection gateway 1 (service) · calls: Chat service
- Connection gateway 2 (service) · calls: Chat service
- Chat service (service) · calls: Message-ID dedup, Message store, Delivery log (per conversation)
- Message-ID dedup (dedupStore) · calls: none
- Message store (kvStore) · calls: none
- Delivery log (per conversation) (queue) · calls: none
- Delivery workers (worker) · calls: Message store
Requirements
Functional
- Send a message to a user or a group (up to a few hundred members).
- Deliver in real time to online devices; store and deliver later to offline ones (with a push notification).
- Per-conversation message order; sent / delivered / read receipts.
- Fetch conversation history, paginated.
Non-functional
- End-to-end delivery latency under ~200 ms p99 between online users in a region.
- No message lost once the sender sees 'sent'; no message shown twice.
- Hundreds of millions of concurrent connections.
Worked estimates
Messages
≈ 230k messages/s average, ≈ 500k/s peak
500M DAU × 40 messages/day ÷ 86,400
Storage
≈ 4 TB/day, ≈ 1.5 PB/year before replication (if the server keeps history)
20B messages/day × ~200 B (body, IDs, timestamps)
Connections
≈ 200–400 gateway hosts; memory per idle connection (tens of KB) is the limit
200M concurrent devices ÷ ~1M connections per tuned gateway host
Group fan-out
delivery work is several times the send rate
average group 20 members × group share of traffic
Bandwidth
≈ 200 MB/s ingress; small, connection count dominates
500k/s × 400 B including headers
API
WS send { conversationId, messageId, body }
Request: messageId is generated by the client (UUID) and reused on retry
Response: ack { messageId, serverSeq } once durably stored
WS ← message { conversationId, serverSeq, ... }
Request: pushed by the server
Response: client acks delivery; server emits a 'delivered' receipt
GET /v1/conversations/{id}/messages?before=seq
Request: cursor by sequence number
Response: 200 { messages[] }
Data model
- messages — store: msgstore
- inbox — store: msgstore
messages: partition key conversationId, clustering key serverSeq (or a time-ordered ID), so history is one partition range read — the layout Discord describes for Cassandra.
inbox: per-user pending deliveries for offline devices, deleted once acknowledged.
A dedup record per (sender, messageId) with a TTL of a day turns client retries into no-ops.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Connections and routing
Each device holds a WebSocket (or MQTT, in Facebook Messenger's case) to a gateway. A presence/session service maps userId → gateway, so a delivery worker knows where to push. Gateways are stateful, so deploys must drain connections gradually and clients reconnect with jittered backoff; a gateway crash otherwise turns into a reconnect storm.
Exactly-once from the user's point of view
Networks give at-least-once at best. The client assigns a message ID and retries until acknowledged; the server stores (sender, messageId) in a dedup table and treats a repeat as success without re-storing. Delivery to recipients is also at-least-once; the recipient's client dedups by message ID. Together that looks like exactly-once to people.
Ordering
Global order is unnecessary and expensive. Order within a conversation is required: the chat service assigns a per-conversation sequence number when it stores the message (a counter in the conversation's partition), and the delivery log is partitioned by conversation so consumers see messages in that order. Clients display by sequence number and fill gaps by fetching history.
Groups and end-to-end encryption
Small groups fan out on send (one delivery per member device). With end-to-end encryption (the Signal protocol, used by WhatsApp) the server never sees plaintext: clients encrypt per recipient device or with a group sender key, and the server only routes and stores ciphertext — which also means server-side search is impossible.
Failure scenarios
Acknowledgements from the chat service are lost
Scenario and expected result
From t = 1.5 s to 1.58 s every response from the chat service to gateway 1 is dropped. The messages were stored, but the gateway times out after 40 ms and retries.
Expected: The retries carry the same message ID, the dedup store recognises them and the chat service answers 'sent' without storing again. No duplicates. Most senders see a short delay; the few whose three attempts all fell inside the cut get an error, and the device retries later with the same ID — still safe.
Delivery workers crash
Scenario and expected result
All delivery workers die from t = 1.3 s to 1.8 s. Messages they had taken are unacknowledged and reappear after the 300 ms visibility timeout.
Expected: Sending keeps working because it ends at the durable log. Delivery pauses, then catches up; redelivered messages stay in order because the log is partitioned by conversation and a partition is consumed in order. Nothing is lost, nothing is reordered, recipients just see a delay.
The message store slows down
Scenario and expected result
From t = 1.5 s to 2.2 s the message store runs 8× slower (a compaction storm).
Expected: Stores take 12–16 ms and some exceed the gateway's 40 ms timeout, so the gateway retries. Dedup keeps the retries harmless, but they add load to a store that is already slow — retries need a budget, or they turn a slowdown into an outage.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
No idempotency key
- Simpler, but a send whose acknowledgement is lost is retried and stored twice — the recipient sees the message twice. Run the lost-acknowledgement scenario against it.
Reference
- Inspect the architecture above and run the same scenario.
Polling instead of persistent connections
- Plain HTTP and trivial load balancing, but latency equals the poll interval and most polls return nothing: 200M devices polling every 5 s is 40M requests/s of mostly empty responses. Long polling or WebSockets push only when there is something to say.
Reference
- Inspect the architecture above and run the same scenario.
Relational store for messages
- Fine for millions of messages a day, and transactions make sequence assignment easy. At billions per day the write volume and the append-only, partition-local access pattern suit a wide-column store better; Discord moved from MongoDB to Cassandra (and later ScyllaDB) for this reason.
Clarifying questions
- 1:1 only, or groups — how large?
- Do we store history on the server, or only until delivered?
- Multiple devices per user?
- End-to-end encrypted?
- Which receipts are needed?
Common mistakes
- No client-generated message ID, so retries duplicate messages.
- Promising global ordering.
- Polling for new messages at scale.
- Storing messages keyed by user instead of by conversation, so history needs a fan-in.
- Ignoring the reconnect storm after a gateway failure.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Send and receive.
- 2/4 Adds groups or offline delivery.
- 3/4 Ordering, delivery guarantees, latency and scale.
- 4/4 Defines 'sent/delivered/read' precisely and what 'no duplicates' means.
estimates / levels 1-4
- 1/4 None.
- 2/4 Message rate.
- 3/4 Message rate, storage and connection count.
- 4/4 Gateway sizing by memory per connection and group fan-out cost.
api / levels 1-4
- 1/4 REST send.
- 2/4 WebSocket messages.
- 3/4 Client message IDs and acks.
- 4/4 Receipts, history cursors and reconnect/resume semantics.
data model / levels 1-4
- 1/4 Messages table.
- 2/4 Keyed by conversation.
- 3/4 Partitioned by conversation, ordered by sequence; inbox for offline.
- 4/4 Dedup table with TTL; hot-partition handling for huge groups.
high level / levels 1-4
- 1/4 Server relays messages.
- 2/4 Gateways + chat service + store.
- 3/4 Adds delivery log/queue and presence routing.
- 4/4 Explains each failure: gateway crash, store slowdown, worker crash.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions WebSockets.
- 3/4 Idempotent sends and per-conversation order.
- 4/4 Group fan-out, E2E encryption constraints and reconnect storms.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions monitoring.
- 3/4 Delivery lag and connection churn as risks.
- 4/4 SLOs on delivery latency with alerting on consumer lag and reconnect rate.