Case study: YouTube (video streaming)
Design video streaming at scale: resumable chunked uploads, a parallel transcoding DAG for adaptive bitrate segments, and ISP-embedded caches delivering 100 Tbit/s.
35 min9 steps
By the end you can
- Estimate storage growth, transcoding compute, concurrent streams and egress bandwidth from watch time
- Design a chunked upload and parallel transcoding DAG that produces adaptive bitrate segments
- Explain how segmented adaptive bitrate streaming works with CDNs and origin shields
- Predict what happens during a viral premiere, an upload surge or a CDN edge outage
Requirements
Upload, transcode, stream
- Resumable hours-long uploads; adaptive bitrate; seeking; eventual view counts
Delivery dominates
- Startup ~2s; durable originals; ~100 Tbit/s scale assumption
Estimates
Uploads
500 h/min
30k video-hours/wall-clock hour assumption
Originals/day
0.7 PB
30k h/hour / 1 GB/h / 24
Renditions/day
~1 PB
~1.5x originals before redundancy
Concurrent streams
42M
1B watch-hours/day / 24
Egress
~100 Tbit/s
42M / 2.5 Mbit/s
Transcoders
Tens of thousands
Parallel ~4s chunks; workload-dependent
API
POST /upload; PUT /upload/{videoId}/chunk/{n}
- 201 after durable chunk; GET /videos/{id} processing state
GET /v/{id}/master.m3u8 or .mpd
- Manifest / immutable rendition segments /{rendition}/{n}.m4s; long CDN TTL
Data model
Video metadata
- videoId, ownerId, title, status, manifestKey; async view-count aggregation
Media blobs
- Immutable (videoId, rendition, index); retain originals for future codecs
High-level design
Put the steps from uploading a video chunk to a viewer streaming it in order.
- 1The creator uploads a raw chunk to the origin, which stores it in the object store
- 2A transcoder worker encodes the chunk into ~6 renditions and writes the .m4s segments to the object store
- 3Once all chunks finish, the manifest is published and the video status becomes playable
- 4The origin records the chunk in metadata and publishes a job to the transcode queue
- 5A viewer's player fetches the manifest and streams cached .m4s segments from the CDN edge
Deep dive: transcoding DAG and 100 Tbit/s delivery
Transcode DAG
- Split at keyframes / encode chunks/renditions / package / publish; deterministic keys make retries idempotent
Delivery hierarchy
- ISP-embedded caches; off-peak pre-positioning; origin shield coalesces long-tail misses
Alternatives
No shield
- Duplicate edge misses overload origin
Progressive file
- Simple; one bitrate; slow seeking
Per-title encoding
- Extra compute buys lower recurring egress
Failure scenarios you can run
Inject
Viral video: 40%
Observe
Segments remain at CDN edge
Result
Long-tail misses dominate origin work
Inject
CDN failure; 300ms DNS TTL
Observe
~4 Gbit/s offered to 2.5 Gbit/s uplink
Result
Queue grows; playback stalls
Inject
Uploads triple
Observe
Durable chunks still acknowledge
Result
Transcode backlog delays publication; viewing unaffected
Wrap up
Watch
- Rebuffer rate, byte hit ratios, transcode lag
Avoid
- App-server media delivery; monolithic encodes; synchronous view counts; QPS-only sizing
Then score yourself from 1 to 4 on each rubric stage.
Answer this step to continue.