File sync & storage (Dropbox)
Keep files synchronised across a user's devices without re-uploading an entire 1 GB file when one paragraph changes. Files are split into content-addressed chunks (~4 MB or Rabin-fingerprinted variable chunks) stored in an immutable object store, while a transactional metadata database tracks versioned chunk lists and notifies linked devices.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: Sync + Block API 1, Sync + Block API 2
- Sync + Block API 1 (service) · calls: Block SHA-256 index, Metadata cache, Metadata DB (files + versions), Content-addressed block store, Device notification stream
- Sync + Block API 2 (service) · calls: Block SHA-256 index, Metadata cache, Metadata DB (files + versions), Content-addressed block store, Device notification stream
- Block SHA-256 index (dedupStore) · calls: none
- Metadata cache (cache) · calls: none
- Metadata DB (files + versions) (database) · calls: none
- Content-addressed block store (objectStore) · calls: none
- Device notification stream (queue) · calls: none
- Long-poll / push notifiers (worker) · calls: Metadata DB (files + versions)
Requirements
Functional
- Upload, download, and automatically synchronise files and folders across multiple devices per user.
- Delta sync and deduplication: upload and download only modified chunks, and skip chunks whose SHA-256 hash already exists.
- Maintain file version history and handle concurrent offline edits without silently losing work (conflict copies).
Non-functional
- Zero silent data loss or corruption; 99.999999999% (11 nines) block durability via replication / Reed-Solomon erasure coding.
- Low-latency change propagation: linked online devices learn of a file change within a couple of seconds.
- Strict separation between small transactional metadata and multi-exabyte immutable block payloads.
Worked estimates
Active clients
≈ 250M connected sync clients holding long-poll / WebSocket channels
100M DAU × ~2.5 connected devices
Block uploads
≈ 8,700 chunk writes/s average, ~3.5 GB/s ingress before deduplication
500M file edits/day × ~1.5 changed chunks (4 MB max, ~400 KB mean compressed) ÷ 86,400
Storage footprint
≈ 1.5 EB logical, stored across erasure-coded warm/cold storage tiers
700M users × ~3 GB average raw ÷ ~1.4 dedup & compression ratio
Metadata QPS
≈ 50k–100k metadata reads/s, 10k writes/s — served by sharded SQL + cache
file commits + manifest fetches + cursor checks
API
POST /v1/blocks/check → PUT /v1/blocks/{sha256}
Request: list of chunk SHA-256 hashes; upload only the missing ones
Response: 200 { missingHashes[] } then 201 per uploaded block
POST /v1/files/{fileId}/commit
Request: { baseVersion, blockHashes: [sha256, ...] }
Response: 200 { newVersion } · 409 version conflict (create conflicted copy)
GET /v1/files/{fileId}/manifest
Request: sinceVersion cursor
Response: 200 { version, blockHashes[], downloadUrls[] }
Data model
- fileMeta — store: metadb
- blocks — store: blockstore
blocks: immutable, content-addressed by `sha256(chunk)`. Because the key is the cryptographic hash of the content, blocks never change in place and can be cached forever.
fileMeta: maps `(workspaceId, fileId) → currentVersion, [sha256_0, sha256_1, ...]`. Updating a file is an atomic CAS (`WHERE version = baseVersion`) on the chunk list.
Device sync cursors: each workspace is a sequenced log (`seq = 1, 2, ...`); devices long-poll `/changes?cursor=lastSeq` so reconnects simply replay missed metadata entries.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Chunking, content addressing, and write-before-commit ordering
A client splits a file into chunks (4 MB fixed or Rabin content-defined), hashes each chunk with SHA-256, and asks the server which hashes are missing. The client uploads only the missing blocks to the block store, verifies their hashes, and **only after all blocks are durably stored** calls `commit(fileId, baseVersion, [sha256...])` on the metadata DB. This ordering guarantees a committed file version never points to a dangling block.
Optimistic concurrency and conflict files
Unlike Google Docs, arbitrary binary files (.psd, .zip, .docx) cannot always be auto-merged character-by-character. Each commit supplies `baseVersion`. If Alice and Bob both edit `v3` offline, Alice's commit advances the file to `v4`; Bob's commit with `baseVersion = 3` gets `409 Conflict`. Rather than overwriting Alice's work, the client saves Bob's blocks as `report (Bob's conflicted copy).docx` so both versions are preserved.
Cold storage tiering and erasure coding
Most uploaded blocks are rarely read after the initial sync across a user's devices. Keeping 3 full copies of an exabyte is prohibitively expensive (`3×` overhead). Dropbox's Magic Pocket groups immutable blocks into large extents and applies Reed-Solomon erasure coding (e.g. 6 data + 3 parity shards across failure domains = `1.5×` overhead) with cold archival tiers for old versions.
Failure scenarios
Shared team folder receives a burst of syncs
Scenario and expected result
A shared project file is synced and edited heavily, driving 35% of requests to one hot key.
Expected: The metadata cache absorbs repeated manifest reads, and content-addressed block deduplication avoids re-writing identical chunks to the object store.
Metadata database leader fails over
Scenario and expected result
The metadata DB leader crashes from t = 1.5 s to 2.0 s while semi-sync replicas elect a new leader.
Expected: Cached manifest reads continue unaffected. File version commits pause during the ~500 ms failover window and then resume on the promoted replica with zero lost versions thanks to semi-synchronous replication and fencing.
Notification workers crash briefly
Scenario and expected result
The push/long-poll notifier workers crash from t = 1.3 s to 1.9 s.
Expected: File commits and manifest fetches succeed without interruption; change events wait in the durable `syncbus` queue and notify connected devices as soon as workers recover.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
Re-upload the entire file on every edit (no chunking or dedup)
- Uploading a whole file on every save instead of only unseen chunks hammers object storage and saturates upload links. In the simulator, skipping block deduplication and metadata caching pushes write latency and load much higher.
Reference
- Inspect the architecture above and run the same scenario.
Fixed-size 4 MB chunks vs Content-Defined Chunking (Rabin fingerprinting)
- Fixed 4 MB chunks are simple, but inserting 1 byte at the beginning of a 100 MB file shifts every boundary and invalidates all 25 chunks. Content-Defined Chunking (rolling Rabin hash that cuts a chunk boundary when `hash & mask == 0`) re-synchronises boundaries immediately after the insertion so only one chunk changes.
Reference
- Inspect the architecture above and run the same scenario.
Pass block bytes through metadata API servers
- Routing gigabytes of file bytes through transactional metadata servers wastes CPU and NIC capacity. Instead, clients upload/download blocks directly against the block store (or via pre-signed S3/storage URLs) and send only the 32-byte SHA-256 hashes to the metadata service.
Clarifying questions
- Maximum file size and typical file size distribution?
- How should concurrent offline edits to the same file be resolved?
- Do we deduplicate blocks within a single user's account or globally across accounts?
- How long do we retain deleted files and old versions?
Common mistakes
- Uploading and downloading whole files instead of content-addressed chunks.
- Committing the metadata row before all referenced blocks are durably written.
- Streaming multi-gigabyte file uploads through the metadata database servers.
- Using fixed-size chunks when middle-of-file insertions are common, without discussing content-defined chunking.
- Having 250M clients poll the metadata database every second instead of using cursor-based long polling.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Upload and download files.
- 2/4 Sync across devices + history.
- 3/4 Chunk delta sync, deduplication, 11-nines durability, conflict preservation.
- 4/4 Separates metadata ACID requirements from immutable block storage.
estimates / levels 1-4
- 1/4 None.
- 2/4 Total storage.
- 3/4 Chunk write rate, dedup savings, metadata QPS, and connected long-poll clients.
- 4/4 Shows why metadata traffic and block bandwidth must be scaled on separate paths.
api / levels 1-4
- 1/4 PUT/GET file.
- 2/4 Chunk upload + metadata.
- 3/4 Pre-check missing hashes → upload blocks → CAS commit manifest.
- 4/4 Cursor-based `/changes` long-polling and pre-signed block URLs.
data model / levels 1-4
- 1/4 Files table.
- 2/4 Files + chunks.
- 3/4 Content-addressed immutable blocks (`sha256`) + versioned manifest rows.
- 4/4 Workspace change sequence log and reference counting / GC for orphaned blocks.
high level / levels 1-4
- 1/4 App server + S3.
- 2/4 Metadata DB + block store.
- 3/4 Separate block plane and metadata plane + async notification bus.
- 4/4 CDN/direct block transfer, cache invalidation, and multi-zone erasure coding.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Explains 4 MB chunks.
- 3/4 Rabin content-defined chunking and write-blocks-before-commit invariant.
- 4/4 CAS conflict copies, long-poll cursor recovery, and Reed-Solomon cold storage.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions durability.
- 3/4 Dangling blocks, silent corruption, and reconnect storms as top risks.
- 4/4 End-to-end checksum scrubbing, orphan block GC watermarks, and failover drills.