Object storage: blobs, durability and cost
Choose a blob layout, distinguish durability from availability, calculate erasure-coding overhead, and budget storage and egress.
18 min7 steps
By the end you can
- Choose an object key and metadata pattern for immutable blobs
- Distinguish durability from request availability
- Calculate replication and k+m erasure-coding storage overhead
- Estimate storage and egress costs with explicit units and assumptions
Store bytes by key
An object holds bytes plus metadata under a key in a bucket. Keep searchable application metadata in a database. Prefer a new key for each blob version. Uploading bytes and publishing a database pointer are separate operations: reconcile orphan uploads and dangling pointers.
Kept safely does not mean reachable now
Durability
- Will the stored object survive?
- Loss is permanent
Availability
- Can a request succeed now?
- An outage may be temporary
A network outage can make intact objects unavailable. Permanent loss harms durability even if other requests succeed. Durability estimates, availability targets and request measurements have different denominators and time horizons; do not convert one into the other.
Three copies cost three times the bytes
Logical data
100 GB
Before replication
Stored bytes
300 GB
3 complete copies
Ideal tolerance
2 copy losses
One intact reachable copy remains
Replication stores complete copies. Three copies have 3x raw storage overhead. Placement matters: three copies on one rack do not survive a rack loss. Repair must restore redundancy before further failures exhaust it.
Recover from a subset of fragments
Fragments
9
Any 6 suffice for this ideal code
Overhead
1.5x
(6 + 3) / 6
100 GB logical
150 GB raw
Up to 3 fragment erasures per group
An ideal maximum-distance-separable code reconstructs from any k of k+m fragments. Raw overhead is (k+m)/k. It tolerates m fragment erasures per group, not arbitrary failures of m regions. Reconstruction and repair consume compute and bandwidth. Placement still determines which failures destroy too many fragments together.
Delivery can cost more than retention
Storage
USD 20
1,000 GB-month x USD 0.02
Egress
USD 450
5,000 GB x USD 0.09
Subtotal
USD 470
Requests and retrieval excluded
Separate stored GB-month from transferred GB. Use provider-billed storage: do not multiply by internal redundancy again when included in the rate. Download frequency and cache misses drive egress. These prices are teaching inputs, not current market rates.
Choose a layout for the workload
Full replicas
- Read from a complete copy
- Higher raw byte overhead
Erasure coding
- Lower raw byte overhead
- Reconstruction when fragments are missing
Both need
- Failure-domain placement
- Repair capacity
- Measured availability
A cold archive may favor coding efficiency; a small hot object may favor simpler reads. Neither choice alone proves a durability target. Test degraded reads, repair load and failure-domain placement against the workload.
Check your mastery
Mastery
80%
4 of 5 fresh questions
Try a fresh variant, then score at least 80% on the mastery check. Wrong answers link back to the teaching step.
Loading question…
Mastery check
5 questions on fresh variants. One attempt each, no time limit. Score 80% or more to master the lesson.
Answer this step to continue.