Case study: File sync & storage (Dropbox)
Design a cloud file sync service: content-addressed chunking, SHA-256 deduplication, write-blocks-before-commit metadata ordering, and conflict copies.
35 min9 steps
By the end you can
- Estimate block ingress bandwidth, deduplication savings, metadata QPS, and long-poll client connections
- Explain fixed vs Rabin content-defined chunking and why blocks are addressed by their SHA-256 hash
- Separate immutable block storage from ACID versioned metadata and enforce write-blocks-before-commit
- Predict how the system handles shared-folder hotspots, metadata DB failovers, and offline edit conflicts
Pin down what you are building
Features
- Delta sync, dedup, version history, offline conflict copies
Targets
- 1 KB-50+ GB files; propagate within seconds
Safety
- 11-nines block durability; end-to-end checksums; no silent overwrite
Work out the numbers
Devices
250M
100M daily users * 2.5
Chunk writes
8.7k/s
500M saves * 1.5 / 86,400
Ingress
3.5 GB/s
400 KB/chunk before 25-35% dedup savings
Physical storage
1.5 EB
700M * 3 GB / 1.4; before redundancy
Metadata
50-100k reads/s
10k writes/s; 250 B manifests
Design the sync protocol and API
Check
POST /v1/blocks/check {hashes} -> missing hashes
Upload
PUT /v1/blocks/{sha256}; verify content checksum
Commit
POST /v1/files/{id}/commit; baseVersion + blockHashes; CAS or 409
Sync
GET /v1/changes?cursor= then manifest; download missing hashes
Put the steps of saving an edited file in the exact order that prevents dangling block pointers.
- 1Client uploads only the missing chunks to the immutable block store and receives 201
- 2Client commits (fileId, baseVersion, blockHashes[]) in an ACID transaction on the metadata DB
- 3Notification stream wakes the user's other devices to fetch the new manifest and missing blocks
- 4Client splits the modified file into chunks and computes each chunk's SHA-256 hash
- 5Client asks the block index which SHA-256 hashes are not yet stored
Model the data
Blocks
SHA-256 -> immutable bytes
fileMeta
workspaceId + fileId -> version, path, blockHashes
workspaceLog
workspaceId + seq -> fileId, newVersion; device persists cursor
Draw the high-level architecture
Deep dive: Rabin chunking, conflict copies, and Magic Pocket
Fixed 4 MB chunks
- 10-byte prefix insert changes every hash
Content-defined chunks
- Rabin rolling boundaries realign; preserve later chunk hashes
Offline conflict
- baseVersion 7 loses CAS to v8; preserve conflict copy
Cold storage
- 6 data + 3 parity shards: 1.5x overhead, tolerates any 3 shard losses
Evaluate alternatives
dropbox-whole-file
- No dedup/cache; goodput <85%
User-held encryption
- Privacy; different user keys prevent cross-user dedup
Pre-signed URLs
- Clients stream blocks directly; metadata API handles manifests
Run the failure scenarios
Inject
35% traffic targets one file
Observe
Cache manifests; skip duplicate blocks
Recover
>99% goodput
Inject
Metadata DB down 500 ms
Observe
Cached reads continue; commits pause
Recover
semiSync + fencing: no lost versions
Inject
Notifiers down 600 ms
Observe
syncbus buffers changes
Recover
Commits/reads >99% goodput
Wrap up and self-assess
Transfer
Content-defined chunks + SHA-256
Commit
Blocks before metadata; CAS + conflict copies
Retain
Version history; erasure-coded cold storage
Answer this step to continue.