Migrate a Production AI Stack Without a Flag Day

Start by drawing the source and target as behavior and state contracts rather than boxes. Enumerate synchronous APIs, streaming events, schemas, vector and cache formats, model inputs and outputs, tool calls, policy decisions, batch jobs, human queues, dashboards, alerts, and recovery procedures. Identify every reader and writer, including scripts and analytical jobs that are absent from request traces. Freeze the baseline workload and evaluation contract so a target cannot appear equivalent merely because the question changed.

1. 01ExpandAdd target interfaces, optional fields, adapters, versioned state, observability, and target capacity while preserving all source behavior. Prove mixed-version operation before sending target traffic.
2. 02MigrateDual-write or transform replayably, backfill from a stable checkpoint, shadow representative reads, reconcile state, and move bounded cohorts or traffic only when compatibility and comparison gates pass.
3. 03VerifyAt the declared target ceiling, observe complete windows, detect unknown consumers, reconcile state and outcomes, exercise rollback, and retain the source while evidence accumulates.
4. 04Contract separatelyAfter explicit approval, remove source reads, writes, routes, state, credentials, dashboards, and support in small reviewable changes. Verify each removal; do not smuggle destructive cleanup into the traffic migration.

Shadow traffic is safe only when the target's effects are isolated. Reuse recorded inputs or mirror sanitized requests with stable identity, but suppress external writes, messages, billing, user-visible output, and tool actions unless a purpose-built sandbox captures them. Compare the right semantics: exact text equality may be meaningless for generative output, while matching availability can conceal quality or safety regression. Use task rubrics, calibrated evaluators, critical slices, deterministic fields, structured side-effect plans, latency, cost, and human review where consequence warrants it.

• Give every source event, backfill record, dual write, comparison, and retry a stable identity so duplicates and omissions are observable.
• Record transform and comparator revisions; changing either invalidates prior reconciliation evidence.
• Separate source-versus-target error and latency from shared dependency failures so rollback decisions have a causal basis.
• Preserve privacy, access, retention, and purpose constraints when copying or replaying production-derived data.
• Define how to compensate or manually reconcile any side effect that cannot be made idempotent or reversed.

The reference planner consumes invented aggregate evidence. Frozen records bind interface, data, evaluation, routing, rollback, and fixture identities to exact source and target revisions. It distinguishes compatibility and readiness gaps that hold from measured agreement, error, or latency regressions that roll exposed traffic back. If regression is present but source capacity or the rollback route is unavailable, it returns `STOP_AND_ESCALATE` and does not pretend traffic can be restored. It caps every advance, turns the traffic ceiling into an explicit verify transition, and refuses to authorize contraction without full traffic, verification windows, retained rollback evidence, and a separate approval.

1def plan_migration(contract: MigrationContract, evidence: MigrationEvidence) -> MigrationPlan:
2    contract = validate_record(contract, MigrationContract)
3    evidence = validate_record(evidence, MigrationEvidence)
4    if evidence.scope != contract.scope or evidence.contract_content_id != contract.content_id:
5        raise ValueError("evidence belongs to another migration contract")
6 
7    # Regressions roll back only while the source route remains viable;
8    # otherwise stop and escalate. Readiness gaps hold, passing evidence
9    # advances once, and contraction remains separately authorized.

example=illustrative_only
decision=ADVANCE_BOUNDED_TRAFFIC
phase=MIGRATE;source=ai-stack-v4;target=ai-stack-v5
target_traffic=0.25->0.50
claim=LOCAL_MIGRATION_PLAN_NOT_EXECUTION_OR_PRODUCTION_CERTIFICATION

Rollback triggers should be observable and tied to the target: semantic regression on a critical slice, error-rate increase, tail-latency ratio, cost or capacity loss, reconciliation divergence, safety incident, or control failure. Decide whether recovery means route-only rollback, source-of-truth restoration, compensating writes, replay, or incident containment. A dual-write bug can corrupt both paths; keeping the source process alive does not undo that state.

Full target traffic begins verification; it does not prove that old readers, offline jobs, rollback data, dashboards, or human procedures are gone. Hold through a declared observation window, reconcile source and target, scan access and dependency evidence, exercise recovery, and obtain explicit contraction approval. Then retire one dependency at a time. Preserve decision records, migration manifests, transform versions, reconciliation results, incidents, and final state proofs even after temporary dual-run telemetry is removed.

1. 01Prove target authorityConfirm all authorized traffic and state ownership use the target while quality, safety, reliability, latency, and cost remain inside the declared envelope.
2. 02Prove consumer exitCombine repository, runtime, access, event, job, and owner evidence; one quiet log window is not proof that a monthly or disaster-recovery consumer migrated.
3. 03Authorize contractionRecord accountable approval, residual risk, retention boundary, recovery alternative, sequencing, stop conditions, and the exact old capability each change removes.
4. 04Verify every removalRemove routing, writes, reads, storage, credentials, capacity, alerts, code, and contracts in bounded steps with observation between them; archive the final evidence ledger.