Build a chaos program that produces evidence instead of theater

• Start from incident history and critical business journeys, not a catalog of failure injectors. Define the steady-state hypothesis, acceptable customer risk, abort authority, and evidence needed to change confidence.

• Map experiments from component to cell, region, control plane, and people-process scope. Select a cohort large enough to expose real dependencies but small enough to stay within an agreed risk budget.

• Experiments must test a falsifiable resilience claim, use production-realistic load and state, protect customers, capture results, and assign remediation ownership. Repeated known failures without action are not learning.

• Provide a guarded experiment control plane with scoped targets, prechecks, automatic abort signals, immutable logs, and kill authority independent of the tested system. Use cells and canary tenants to bound exposure.

• An experiment record links hypothesis, target, blast radius, telemetry, start and stop conditions, observed outcome, and follow-up work. Findings enter the same prioritized reliability portfolio as incidents.

• Stateful experiments need invariants and reconciliation checks after apparent availability recovers. Success includes proving no silent data loss, duplicate effects, or policy bypass occurred.

• Design for the injector or abort channel failing, concurrent real incidents, and misleading green metrics. Stop automatically on customer SLO or safety thresholds and preserve manual out-of-band termination.

• Use least-privilege experiment capabilities, protected environments and tenants, approval by risk class, and strong audit. Never let a chaos tool become a general production remote-execution system.

• Observe customer journey, dependency saturation, control behavior, data invariants, and human detection and response. Compare predicted versus actual detection, containment, and recovery times.

• Fund experiments by expected risk reduction and reuse test harnesses in deployment and disaster-recovery validation. Account for engineer time to remediate; unfunded findings accumulate risk rather than reduce it.

• Staging is safer but misses production scale and dependency reality; production provides truth at customer risk. Progress from replicas and cells to bounded production only when abort controls and observability are proven.

• Begin with incident-derived tests in isolated production cells, automate regression experiments, then add cross-region and socio-technical game days. Retire experiments whose claims are covered by continuous verification.

• Distinguished candidates create a learning and funding system, not just tooling. They connect hypotheses to business journeys, data correctness, human operations, and measurable confidence over time.

Interviewer evaluation rubric