Partitioning and consistent hashing
Choose a shard key, estimate key movement when a ring grows, and explain why balanced storage does not eliminate hot-key load.
26 min8 steps
By the end you can
- Choose range or hash partitioning and calculate an indivisible hot key's load floor
- Estimate moved keys when an equal node joins an ideally balanced ring
- Route a key to its clockwise owner and explain virtual-node trade-offs
- Order a migration that preserves reads and writes during ownership changes
A shard key chooses where work lands
Partitioning splits data; replication copies it. The shard key sets routing, locality and cross-shard work. Start from queries, not hash functions.
Tenant ID
- Tenant queries stay local
- Huge tenant dominates a shard
Random row ID
- Writes spread evenly
- Range queries hit every shard
Constant key
- All work on one shard
Range and hash partitioning fail differently
Range
- Cheap range scans
- Timestamp key: newest range is hot
Hash
- Even spread of keys
- Time-range query scatters
Hash tenant + range time
- Only if the workload needs it
Follow the ring clockwise
hash(key) % N remaps most keys when N changes. On a ring, a key belongs to the first token clockwise, wrapping at the end. Virtual nodes smooth the intervals. Predict what moves, then add and remove nodes.
Estimate how much data must move
Adding a node to N balanced nodes moves about 1 / (N + 1) of keys. This is an expectation, not a guarantee on a small ring; measure bytes too.
Share that moves
1/5
Keys moved
≈ 2M
Modulo routing
≈ 80%
for contrast
Loading question…
Balanced keys can still mean unbalanced traffic
Virtual nodes spread intervals, not requests for one key.
Loading question…
Ownership changes need a data protocol
Changing the routing map copies no data. Use a versioned map, a catch-up path and a coordinated handoff. Rate-limit copying.
1
Copy snapshot
2
Catch up changes
3
Switch owner (versioned map)
4
Retire old copy after rollback window
Order a simplified coordinated shard handoff.
- 1Catch up changes and coordinate the final write handoff
- 2Publish the new ownership version and handle stale clients
- 3Copy a snapshot while tracking subsequent source changes
- 4Retire the old copy only after the safety and rollback window
Partitioning moves coordination into the design
Sharding moves coordination into your design: cross-shard uniqueness, joins and transactions cost more. Pick the smallest layout that meets measured need, and watch per-shard latency and key popularity.
Check your mastery, then create a hot shard
Score 80% or more. Compare the ideal movement with the finite ring and explain the gap.
Mastery check
5 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
In the sandbox, compare uniform keys with a hot key and add capacity: the hot owner stays the bottleneck.
Answer this step to continue.