Migrate a multi-terabyte database without downtime or wishful dual writes

• Define the invariant set, tolerated cutover pause, rollback window, data retention, and whether the new engine changes transaction semantics. Identify tables and operations that can migrate independently rather than treating 12 TB as one atomic move.

• Assume 12 TB, 90k writes/second, 30% annual growth, and a copy channel capped below the bandwidth that would hurt production. Estimate snapshot duration, CDC log retention, transform throughput, and the lag required before cutover.

• No acknowledged write may disappear, transformed data must satisfy business invariants, customer impact must stay within the SLO, and every phase must be resumable and auditable. Cleanup waits until the rollback window expires.

• Take a consistent source snapshot at log position L, bulk-copy through idempotent transformers, and concurrently stream changes after L through CDC. A validation service compares counts, checksums, invariants, and sampled query results; a migration controller owns phases and gates.

• Backfill in stable key ranges with checkpoints and rate limits. CDC events carry source position and operation identity, the destination upserts idempotently, and per-range high-water marks let validators know when snapshot plus deltas are complete.

• Keep the source as the only write authority until cutover; avoid application dual writes where partial success creates ambiguity. During a brief final drain, stop new writes, wait for CDC lag to reach zero, record the terminal source position, switch routing, and reopen.

• Workers retry ranges idempotently, poison transformations quarantine records without hiding the gap, and CDC retention alarms well before data expires. Before cutover, rollback means discard or reset destination state; after cutover, rollback requires reverse replication or a clearly bounded source freeze strategy.

• Encrypt transfer and temporary files, preserve tenant and row-level authorization semantics, and prevent logs from leaking sensitive values. Ensure the destination inherits deletion, residency, retention, and audit obligations before production traffic.

• Track copied bytes and rows, range retries, transform errors, CDC lag, invariant mismatches, destination query divergence, source impact, and cutover gates. Publish a migration dashboard with go/no-go owners and automated thresholds.

• Budget temporary double storage, CDC retention, validation reads, and network egress. Throttle against production latency and schedule large scans on replicas when their lag and failover role allow it.

• Application dual writes reduce cutover pause but create a two-system correctness problem on every write. Snapshot plus CDC keeps one authority and concentrates complexity in a recoverable pipeline; accept a bounded drain if it materially simplifies proof.

• Use expand-contract schema changes, migrate low-risk tables or tenants first, shadow reads and compare, then canary a small traffic slice. After full cutover, retain source read-only through the rollback window, reconcile again, then remove CDC and old schema deliberately.

• A strong answer treats migration as a state machine with evidence, not a copy command. Name the authority at every phase, the exact cutover condition, how mismatches are repaired, and why rollback differs before and after writes begin on the destination.

Interviewer evaluation rubric