Rebalance stateful shards while live traffic keeps mutating them

• Define the shard’s write authority, maximum tolerated pause, acceptable replication lag, and whether clients can retry across an ownership change. Ask which state is derivable and which must be transferred exactly.

• Assume multi-terabyte shards, sustained writes during copy, and network headroom limited to a safe fraction of regional bandwidth. Model move duration and log growth together; a snapshot that copies slower than mutations accumulate never converges.

• Maintain a single fenced writer, preserve acknowledged writes, bound request disruption, verify state before promotion, and allow the controller to resume safely after crashing at any step.

• A strongly consistent placement controller issues monotonically increasing ownership epochs. The destination copies a snapshot, tails the source change log, catches up, then promotion atomically changes routing and fences the source before the destination serves writes.

• Record a snapshot position, copy checksummed ranges, stream later mutations idempotently, and maintain a high-water mark. After routing cutover, forward or reject straggler requests carrying the old epoch rather than applying them locally.

• Every mutating request and storage commit includes the current ownership epoch. Promotion requires a conditional compare-and-swap in the placement store; no amount of wall-clock timing can substitute for fencing.

• Make each move phase durable and idempotent: planned, snapshotting, catching up, fenced, promoted, verified, cleaned. A failure before fencing restarts or abandons the destination; after fencing, rollback requires a new epoch and reverse catch-up, never re-enabling the old writer casually.

• Authenticate the controller and transfer channel, encrypt state in motion, and scope operators to particular clusters and tenants. Cleanup must honor retention and ensure abandoned destination copies do not become forgotten sensitive-data replicas.

• Expose bytes copied, mutation lag, checksum mismatch, throttling, epoch rejects, request redirects, and phase duration per move. Alert when a move cannot converge or consumes its network error budget.

• Throttle copies against foreground latency and reserve temporary double storage. Schedule moves in parallel only across independent failure domains, and use virtual shards to make future movements smaller.

• Stop-the-world moves are simpler but violate availability at large sizes. Dual-write moves seem faster but create a difficult equality problem; snapshot-plus-log with fenced ownership provides one authoritative write path throughout.

• Add epochs and request fencing before automating any movement. Rehearse on read-only or replica moves, compare checksums, then enable small production shards and progressively increase size and concurrency.

• The decisive signals are a monotonic authority epoch, durable move state machine, foreground throttling, and honest rollback after promotion. Principal candidates also discuss controller failure and future shard granularity.

Interviewer evaluation rubric