Pastebin
Store text snippets and serve them by ID. It looks like the URL shortener, but the payload is 20× bigger, so the real design question is where the bytes live: metadata in a database, content in an object store, hot pastes in a cache.
Component labels and connections
- Clients (client) · calls: Load balancer
- Load balancer (loadBalancer) · calls: API 1, API 2
- API 1 (service) · calls: Paste cache, Metadata DB, Object store
- API 2 (service) · calls: Paste cache, Metadata DB, Object store
- Paste cache (cache) · calls: none
- Metadata DB (database) · calls: none
- Object store (objectStore) · calls: none
Requirements
Functional
- Create a paste (text up to 10 MB, typically ~10 KB) and get back an unguessable ID.
- Read a paste by ID; optional syntax highlighting and raw view.
- Pastes can expire (10 minutes to never) or be burn-after-read.
- Signed-in users can list and delete their pastes.
Non-functional
- Reads p99 under 200 ms for typical pastes.
- 99.9% availability for reads; a paste, once acknowledged, is never lost.
- Storage cost dominates, so cold content must sit in cheap storage.
Worked estimates
Write rate
≈ 116 writes/s average, ≈ 230/s peak
10M new pastes/day ÷ 86,400 s
Read rate
≈ 1,160 reads/s average, ≈ 2,300/s peak
10 reads per write
Content storage
≈ 183 TB — object store territory (≈ $4k/month at $0.023/GB)
10M × 10 KB/day × 365 × 5 years
Metadata storage
≈ 5.5 TB — a sharded relational or KV store
18B rows × ~300 B
Read bandwidth
≈ 23 MB/s (≈ 190 Mbit/s) of egress at peak
2,300/s × 10 KB
Cache
≈ 200 GB; fewer entries than the URL shortener, but 20× the bytes each
hottest 20% of a day's reads: 0.2 × 100M × 10 KB
API
POST /v1/pastes
Request: { content, syntax?, expiresIn?, burnAfterRead? } + Idempotency-Key
Response: 201 { id, url } · 413 too large · 429
GET /p/{id}
Request: —
Response: 200 text · 404 · 410 expired
DELETE /v1/pastes/{id}
Request: auth: owner
Response: 204
Deletes metadata, schedules the blob for deletion and invalidates the cache.
Data model
- pastes — store: db
- blobs — store: objects
pastes holds small metadata rows and a pointer (blobKey) to the content.
Content lives in an object store keyed by blobKey: cheap per GB, very durable, but tens of milliseconds per read.
byExpiry lets the cleanup job find expired pastes without a full scan.
High-level design
Trace permitted calls in the architecture above, then open the reference canvas to inspect routing.
Deep dives
Metadata and content in different stores
Small, frequently queried fields (owner, expiry, syntax) go in a database that supports indexes. Large immutable content goes in an object store, which is far cheaper per GB and designed for 11 nines of durability, but slower per request. Write the blob first and the metadata second: a crash between the two leaves an unreferenced blob that a sweeper deletes, whereas the opposite order leaves a paste that 404s. Because pastes are immutable, the cache never needs invalidating except on delete or expiry.
Expiry, burn-after-read and cleanup
Check expiresAt on read so expired pastes return 410 immediately. A background job walks the byExpiry index in small batches, deletes metadata and then blobs. Burn-after-read needs an atomic read-and-delete — a conditional delete on the metadata row — or two concurrent readers can both see it. Object stores also offer lifecycle rules that delete by age, which removes most of the sweeper's work for fixed TTLs.
IDs and abuse
Use random 8-character base-62 IDs (2 × 10¹⁴ values), not a counter: pastes are often secrets, so IDs must not be enumerable. Rate-limit creation per IP and account, cap size at the gateway (413 before the body is read), and scan for leaked credentials — services such as GitHub's secret scanning partner programme exist because pastes leak keys.
Failure scenarios
The object store slows down 8×
Scenario and expected result
From t = 1.5 s to 2.3 s object-store reads take about 120 ms instead of 15 ms — a degraded storage tier.
Expected: Cache hits are untouched, while misses and new pastes slow down. API workers spend longer waiting on the store, so their utilisation rises: a slow dependency consumes your concurrency even though it is not your CPU.
Packet loss between one API server and the object store
Scenario and expected result
From t = 1.2 s to 2.4 s, 10% of packets on the API 1 → object store link are dropped. A lost packet looks like a timeout to the caller.
Expected: Blob reads retry with exponential backoff and jitter, so almost every read still succeeds; the cost is a longer tail. Paste creation does not retry (the blob write is not behind an idempotency key here), so a few creates fail.
A paste goes viral
Scenario and expected result
One paste receives 40% of all reads — a leaked config file linked from a popular forum.
Expected: The viral paste is served from the cache after its first read. The remaining traffic is spread over the long tail, so the object store sees more misses — the hot key is the cheap part.
Alternative designs
Reference
- Inspect the architecture above and run the same scenario.
Keep the content in the database
- One system, one transaction, no orphaned blobs. But 183 TB of text in a relational store means many more shards, slower backups and replicas that copy every byte. Below a few TB it is a reasonable choice; at this scale the object store wins on cost and operational load.
Reference
- Inspect the architecture above and run the same scenario.
No retries on blob reads
- Object-store reads are idempotent, so retrying with backoff is safe and cheap. Without retries, every lost packet between the API and the store becomes a failed read.
Reference
- Inspect the architecture above and run the same scenario.
A CDN in front of public pastes
- Public pastes are immutable, so they cache perfectly at the edge and the origin sees only first reads. Private and burn-after-read pastes must bypass the CDN, and deletes need a purge call.
Clarifying questions
- What are the typical and maximum paste sizes?
- Are pastes public, unlisted or private? Do IDs need to be unguessable?
- What expiry options exist, and is burn-after-read required?
- How long must pastes be kept if they never expire?
Common mistakes
- Putting multi-terabyte content in the relational database by default.
- Sizing the cache by entry count instead of bytes.
- Writing metadata before the blob, which can leave pastes that 404.
- Sequential IDs for content that is often secret.
- Retrying non-idempotent writes without an idempotency key.
Self-assessment rubric
requirements / levels 1-4
- 1/4 Create and read only.
- 2/4 Adds expiry or size limits.
- 3/4 States size distribution, read/write ratio and durability.
- 4/4 Also covers burn-after-read semantics, privacy of IDs and deletion.
estimates / levels 1-4
- 1/4 No numbers.
- 2/4 QPS only.
- 3/4 QPS, content vs metadata storage and bandwidth.
- 4/4 Uses storage numbers to justify the object store and sizes the cache in bytes, not entries.
api / levels 1-4
- 1/4 Vague endpoints.
- 2/4 Create/read with status codes.
- 3/4 Adds 413/410/429 and idempotency.
- 4/4 Explains how delete propagates to blob, cache and CDN.
data model / levels 1-4
- 1/4 Content and metadata undifferentiated.
- 2/4 One table holding content.
- 3/4 Metadata table with a blob pointer and owner index.
- 4/4 Adds the expiry index and justifies write ordering between stores.
high level / levels 1-4
- 1/4 Single box.
- 2/4 LB, API, database.
- 3/4 Adds object store and cache on the read path.
- 4/4 Explains which tier fails how, and that cache hits shield the slow store.
deep dive / levels 1-4
- 1/4 None.
- 2/4 Mentions TTLs.
- 3/4 Designs expiry cleanup and ID generation.
- 4/4 Handles burn-after-read atomically and analyses retries vs idempotency for each step.
wrap-up / levels 1-4
- 1/4 Stops.
- 2/4 Mentions monitoring.
- 3/4 Lists bottlenecks and mitigations.
- 4/4 Prioritises cost, abuse and the slow-dependency risk with concrete signals.