Storage
Object Storage with Durability Guarantees
S3-style object store with 11 nines of durability and exabyte scale.
Scale to anchor on
Exabytes of data, trillions of objects, 11 nines durability, regional and cross-region replication.
Requirements
Functional
- PUT/GET/DELETE by key within a bucket.
- Multipart upload for large objects.
- Versioning and lifecycle policies.
- Strong read-after-write within a region.
Non-functional
- Extremely high durability.
- High throughput.
- Cost-tiered storage classes.
High-level architecture
Front-end load balancers route to an index layer that maps bucket+key to placement. Erasure coding chunks each object across many disks and racks. Background scrubbers detect bit rot and rebuild. Lifecycle workers move cold objects to cheaper tiers.
Components
Index service
Bucket+key → placement metadata; sharded and replicated.
Storage nodes
Hold erasure-coded data chunks on commodity disks.
Scrubber service
Background integrity checks; rebuilds from parity on detection.
Lifecycle workers
Move objects across tiers per bucket policy.
Key decisions
Erasure coding over plain replication.
At exabyte scale, 3x replication is 200% overhead; erasure coding cuts that to 30–60% with comparable durability.
Multipart upload.
Single large PUT cannot reliably traverse the internet; multipart enables retry and parallelism.
Index layer separate from storage.
Index is hot, small, and consistency-sensitive; storage is cold, large, and durability-sensitive — different scaling shapes.
Strong read-after-write per region.
Achieved by index-level coordination; cross-region remains eventual.
Pitfalls
- Treating durability as just replica count — bit rot is the real enemy.
- Centralized index without sharding.
- Ignoring the metadata cost of trillions of small objects.
- Lifecycle policy that surprises customers during a billing audit.
Follow-up questions
- How do you achieve 11 nines durability concretely?
- How does erasure coding work and how do you tune it?
- How does cross-region replication interact with consistency?