Design safe rollback across application, schema, and event contracts

• Classify every change as backward compatible, forward compatible, reversible, compensatable, or irreversible. Define the rollback window and which external consumers and jobs may observe the new contract.

• Model mixed-version duration, event retention, backfill progress, and the volume of writes created by the new path. A rollback plan must handle all states already produced at the current rollout percentage.

• Old and new binaries coexist safely, schema changes use expand-contract, events remain consumable, and operators can stop effects independently. Irreversible actions require a forward repair rather than fictional rollback.

• Decouple deploy from activation with server-side flags, dual readers, versioned event envelopes, and additive schema. A release manifest records component compatibility, migrations, flags, consumers, and approved rollback or roll-forward actions.

• Writers emit the oldest compatible representation until consumer readiness is proven, or dual-publish to distinct versioned topics. Backfills include transform version and before-image or recomputation source when reversal is required.

• Schema contraction occurs only after all writers and readers have left the old field. Rollback may disable new decisions immediately while preserving additive data that old code safely ignores.

• Stop rollout and side effects first, then choose binary rollback, flag disablement, compensating job, or forward fix. Validate mixed states and prevent automated cleanup from deleting compatibility fields during the incident.

• Rollback cannot revive a known-vulnerable path or weaker authorization contract. Feature and migration controls are privileged, audited, and protected from stale configuration.

• Break down errors and business outcomes by application version, flag cohort, schema version, event version, and backfill state. Predeclare rollback thresholds and verify the rollback itself against SLOs.

• Compatibility and dual publication add temporary storage and compute; budget their lifetime and removal owner. Avoid maintaining indefinite dual paths by attaching explicit contraction gates.

• Backward-compatible releases take more planning but preserve options. Fast destructive migrations save temporary complexity and transfer risk into an incident where recovery is slowest.

• Practice expand, migrate, contract as separate releases. Canary producers after consumers are ready, shadow new reads, and remove old contracts only after retention and rollback windows pass.

• Strong candidates ask what new state already escaped. They treat rollback as a matrix across code, data, events, security, and external consumers, and know when roll-forward is safer.

Interviewer evaluation rubric