Case study: News feed
Design a news feed: fan-out on write versus fan-out on read, a queue that turns load spikes into delay, and idempotent fan-out under at-least-once delivery.
35 min9 steps
By the end you can
- Compute fan-out write amplification and the cost of a celebrity post
- Choose between fan-out on write, fan-out on read and a hybrid from the read/write ratio and follower distribution
- Make fan-out idempotent under at-least-once delivery and paginate a live feed correctly
- Predict how the design behaves under a post surge, a worker crash and a cache flush
Requirements
Features
- Publish; newest-first followed-account feed; pagination
Targets
- Feed p99 <200 ms; posts visible within seconds; reads:posts 100:1
Isolation
- Posting fast during fan-out lag; stale feeds acceptable
Estimates
Feed reads
35k/s
300M * 10 / 86,400; peak 70k/s
Posts
350/s
300M * 0.1 / 86,400
Fan-out
70k inserts/s
350 * 200 followers; 200x amplification
Feed IDs
1.2 TB
300M * 500 * 8 B
Celebrity
12 minutes
50M inserts / 70k/s capacity
API
Post
POST /v1/posts + Idempotency-Key returns 201
Feed
GET /v1/feed?cursor= returns posts + nextCursor
Follow
POST /v1/follows/{userId} returns 204; optional backfill
Page
Last-ID cursor stays stable as new posts arrive
Data model
Posts
Source of truth; shard by author
Feeds
User maps to newest 500 post IDs; hydrate bodies from cache
Follows
Index both directions for fan-out
High-level design: fan-out on write
Order what happens when a user posts.
- 1The worker acknowledges the message
- 2The API publishes one fan-out message and returns 201
- 3A fan-out worker takes the message from the queue
- 4The worker writes the post ID into each follower's feed
- 5The API stores the post in the posts database
Deep dive: push, pull or hybrid
Push
- Posts/s * followers writes; read one key
Pull
- Reads/s * followed accounts lookups; cheap posting
Hybrid
- Push ordinary authors; merge celebrities on read; rebuild inactive feeds
Alternatives you can run
Pull
- 8 queries/read in simulator overwhelm posts DB
Append IDs
- Crash after write before ack duplicates feed item
Idempotent insert
- Unique user/post key tolerates replay
Failure scenarios you can run
Inject
150 to 600 posts/s; workers 400/s
Observe
Queue absorbs excess
Recover
Posting fast; about 1 s lag drains
Inject
Workers down 0.6 s
Observe
Feeds pause; replay in-flight work
Recover
Idempotent writes prevent duplicates
Inject
Feed cache empty
Observe
Precomputed store handles reads
Recover
Cache loss stays cheap
Wrap up
Build
Durable queue; idempotent capped feed IDs
Scale
Celebrity threshold; read-time merge
Watch
Freshness SLO on fan-out lag; hydrate cached bodies
Then score each stage from 1 to 4.
Answer this step to continue.