Partition high-cardinality entities when the hottest keys keep changing

• Identify required ordering and aggregation scope, how quickly heat shifts, and whether one series can be split safely by time or write stripe. Define the cost of tenant scans versus point writes.

• Model p99 series write rate, not only billions of keys, and track bytes, compaction, and query fan-out per partition. Assume a hot key can exceed one node and requires an explicit split strategy.

• Absorb sudden skew, preserve per-series semantics, move ownership without lost writes, and bound query amplification. Placement changes must not oscillate with every brief spike.

• Use many virtual partitions with a consistent placement directory, heat telemetry, and a rate-limited rebalancer. Promote extreme keys to salted write stripes or time buckets, then merge reads through a per-series manifest.

• Writers resolve a cached placement epoch and include it with each batch. Storage rejects stale epochs; the mover transfers immutable segments and tails new writes before publishing the next manifest.

• Ordering is per stripe or time bucket, so a sequencer or merge rule restores the promised per-series view. Tenant-wide queries accept scatter-gather with limits or consume pre-aggregated tenant indexes.

• Fencing prevents old placements from accepting writes after a move. Rebalance controllers use hysteresis, move budgets, and rollback states; a failed split leaves the original owner authoritative.

• Placement and manifests are tenant-scoped, and hot-key metrics avoid exposing customer identifiers. Movement preserves encryption and locality policy, including deletion across obsolete copies.

• Measure skew ratios, partition saturation, placement-cache misses, stale-epoch rejects, move convergence, read fan-out, and rebalancer churn. Alert on heat that no legal split can absorb.

• Virtual partitions and adaptive stripes improve utilization but add metadata and read amplification. Keep cold data compact, reserve burst headroom, and move only when predicted incident cost exceeds transfer and fragmentation cost.

• Pure hashing is simple but cannot split one key; salting scales writes but scatters reads and ordering. Use adaptive salting only for exceptional keys and store a manifest that hides layout from clients.

• Introduce placement epochs and virtual partitions before dynamic splitting. Shadow heat recommendations, manually approve early moves, then automate with conservative hysteresis and post-move verification.

• Principal candidates acknowledge that partition count does not solve a single hot key. They design fencing, adaptive split forms, query recomposition, movement economics, and stability against rebalancer oscillation.

Interviewer evaluation rubric