Failure labAdvanced110 min estimateOriginal publication

AI Incident Response and Correction Loops

Restore safety before certainty: bound exposure, support affected people, preserve the changing decision path, and require every correction to survive as an owned regression gate.

Authorship: InterviewsVector
Published / updated: 2026-09-22 / 2026-09-22
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary sources; they do not claim causal validity, production telemetry, safety certification, live incident response, or regulatory compliance.

The decision in one pass

Respond to an AI incident by reducing harm first, while explicitly preserving uncertainty. Declare an incident against user or system impact, assign severity, incident command, harm ownership, communications, and decision rights, then bound exposure with a kill switch, routing change, permission reduction, human handoff, or feature disablement independent of the affected model path. Preserve versioned evidence for assignment, retrieval, policy, prompts, models, tools, effects, outputs, telemetry health, and user reports with privacy and legal handling. Do not call every failure a model problem: test hypotheses across product, data, retrieval, policy, orchestration, tools, vendors, and operations. Reconcile possible side effects, notify and remediate affected people as required, bind corrections to a new revision and regression gates, canary recovery, observe enough windows, and feed the incident into evals, the safety case, runbooks, ownership, and architecture.

Why this matters

AI incidents are difficult to reproduce because outputs are sampled, dependencies and knowledge change, traces may omit content, and user reports arrive after the relevant versions have moved. Teams can waste the response debating whether the model or application is to blame while harmful exposure continues. A prompt patch may hide one symptom but leave the authorization, retrieval, rollback, or monitoring failure untouched.

You will be able to

Declare and triage AI incidents by user and system impact rather than model novelty.
Contain exposure through deterministic controls while preserving affected-user support and evidence.
Separate observation, hypothesis, contributing condition, root cause, and correction across system layers.
Bind every recovery revision to explicit regression gates, canary evidence, observation windows, and rollback.
Turn incident learning into durable evaluation, safety-case, platform, ownership, and operational changes.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Map impact, affected people, exposure, versions, decision path, authority, dependencies, evidence locations, communications, and recovery constraints.
02
Derive
Derive severity, containment deadlines, evidence obligations, hypothesis layers, correction evidence, recovery gates, observation windows, and closure criteria.
03
Build
Build a deterministic audit of an invented incident snapshot and revision-bound correction plan; do not execute a response action.
04
Stress
Remove evidence, delay containment, omit harm assessment, forge scope, collapse cause layers, reuse the failed revision, and close before recovery evidence matures.
05
Operate
Monitor incident state, exposure, harmed-user remediation, containment efficacy, evidence integrity, regression gates, recovery canary, recurrence, and follow-up ownership.
06
Defend
Defend why containment precedes root-cause certainty, why a prompt patch may be insufficient, and which evidence authorizes recovery and closure.

Contain impact before the causal story is complete

Declare on credible impact or control failure, not only on a confirmed model defect. Severity should reflect scope, reversibility, affected people, data sensitivity, external effects, legal or contractual duties, and the reliability of current containment. Name an incident commander and a separate harm owner so restoring service cannot eclipse user remediation. Preserve decision authority when the model, vendor, or primary control plane is degraded.

Immediate objective	Example action	Evidence to preserve
reduce exposure	disable feature, tool, route, tenant, or revision	who changed what, when, under which authorization
stop external effects	revoke credentials or permissions	pending intents, receipts, retries, and ambiguous outcomes
support affected people	human handoff, correction, notification	impact scope, contact state, remedy, and owner
preserve learning	freeze version and trace manifests	model, prompt, retrieval, policy, tool, data, and telemetry revisions

Separate what happened from why you think it happened

01Write observationsRecord affected outcomes, timestamps, cohorts, versions, traces, user reports, tool receipts, and telemetry gaps without causal language.
02Generate layer-specific hypothesesTest product framing, data, retrieval, policy, prompt, model, tool, orchestration, renderer, vendor, and operator conditions.
03Reconcile effectsIdentify writes or messages that may have committed, stop retries, query authoritative state, and plan corrections or compensation.
04Preserve uncertaintyMark unknown exposure, missing spans, changed dependencies, non-reproducible samples, and evidence that might be incomplete or untrusted.

A model may generate the visible error while the actionable root cause is stale retrieval publication, an absent authorization boundary, an invalid metric, unsafe product affordance, or a rollback path that was never exercised. Root cause is the condition whose correction meaningfully reduces recurrence within the bounded system; contributing factors remain important even when they are not the primary label.

Audit response state and gate recovery

incident_playbook.py

1def audit_incident(contract: IncidentContract, snapshot: IncidentSnapshot) -> IncidentReport:
2    contract = validate_record(contract, IncidentContract)
3    snapshot = validate_record(snapshot, IncidentSnapshot)
4    if snapshot.scope != contract.scope or snapshot.contract_content_id != contract.content_id:
5        raise ValueError("snapshot belongs to another incident contract")
6    # Missing evidence, harm assessment, containment, or required recovery proof escalates.
7    # A contained incident needs a revision-bound plan before recovery is gated.

Expected output

example=illustrative_only
decision=RECOVERY_GATED
violations=0
next_actions=run-regression-gates,canary-recovery-revision
containment_deadline_minutes=60
claim=LOCAL_INCIDENT_AUDIT_NOT_LIVE_RESPONSE

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/ai_incident_response

The invented SEV2 fixture records containment within 25 minutes, a digest-only evidence manifest, user-impact assessment, two containment actions, a retrieval root-cause hypothesis with policy and product contributors, a new recovery revision, and two regression gates. The audit enforces severity-ordered deadlines, frozen evidence, harm ownership, containment records, typed cause layers, revision change, scope and digest binding, exact values, correction gates, and enough recovery observation windows before closure.

The artifact does not detect an incident, validate severity, freeze real evidence, disable a feature, contact affected people, establish legal duties, determine a true root cause, run a gate, or execute recovery. Its next actions are deterministic teaching output over invented data, not a live incident command system.

Triage an AI incident before explaining it

Inspect an illustrative severity, response state, evidence posture, user-impact assessment, and correction plan. Predict the next response decision before revealing the deterministic audit result.

Choose the first defensible incident action

Triage from active harm and evidence, not from a favorite root-cause theory. Contain impact, preserve version-bound records, establish ownership, and turn confirmed escapes into durable gates.

Incident record

Sensitive content appears in tracesAn observability change records raw prompts into a broadly accessible index.A new model increases harmful responsesThe release identity is known and the prior serving version remains healthy.User harm is active but the cause is unclearRetrieval, policy, and model versions all changed inside one release window.Containment succeeded; the escape path is knownA long-tail regression passed because the slice was absent from release evaluation.

People, releases, traces, and harm reports are invented. This lab does not operate a production system or store incident data.

Incident-triage evidence
Signal	Decision contract	Observed evidence
Active harm	Sensitive fields are redacted before export	Raw prompt fields still entering the index
Access	Need-to-know incident access	Index readable by the general analytics role
Evidence	Preserve a scoped forensic copy	Retention snapshot is available
Owner	Privacy incident commander assigned	Commander paged; acknowledgement pending

Synthetic evidence only. The decision remains hidden until you check a prediction.

What action comes first at this incident stage?

Stop capture and restrict exposed dataRollback the implicated releasePause the risky path and preserve evidenceAdd the missing test and durable release gate

Choose an outcome before checking the evidence contract.

Require a new revision to earn recovery

Translate every material finding into the narrowest responsible correction and a test that would have failed before it. A prompt or model change may be part of the fix, but deterministic authorization, retrieval publication, schema validation, rollback, user communication, or ownership failures need changes at their owning layer. Bind the correction plan to the affected revision and name a different recovery revision so the incident cannot close on the same unmodified system.

recover ⇔ containment stable ∧ regression gates pass ∧ canary guardrails pass ∧ rollback ready

Recovery is a new bounded release decision. Keep affected cohorts and delayed harms observable for the declared windows before closure.

Finding	Durable correction	Recovery evidence
stale index reached users	atomic publication and freshness gate	old revision rejected; canary uses expected index
tool exceeded user scope	downstream tenant/resource authorization	adversarial cross-tenant call denied
monitor missed harm	new labeled signal and completeness SLO	historical cases detected without unacceptable noise
rollback failed	independent route and rehearsed runbook	timed exercise restores safe path

Close the response only after ownership moves into normal work

Closure means exposure is controlled, affected people and effects are reconciled, the recovery revision has passed its gates and observation windows, residual risk has an owner, required communication is complete, and follow-up work has accountable dates. Preserve the timeline and uncertainty. A blameless review removes fear from reporting; it does not remove ownership from architecture, controls, and decisions.

Add confirmed and near-miss cases to golden datasets, adversarial suites, and online guardrails with provenance.
Reopen the safety case where a hazard, control, evidence item, assumption, or decision right failed.
Update platform defaults, tool scopes, telemetry schemas, runbooks, support flows, and vendor requirements where the cause was systemic.
Track correction effectiveness and recurrence; completing a ticket is not evidence that risk fell.

Operate at three altitudes

Production lens

— Maintain independently operable kill switches, credential revocation, routing changes, human fallback, version manifests, effect reconciliation, and affected-user support before an incident begins.
— Track time to declare, contain, scope, assess user harm, reconcile effects, gate recovery, notify, close, and complete durable corrections by severity and product boundary.
— Retain bounded evidence with access, integrity, privacy, legal, and deletion controls; missing telemetry and unknown exposure remain explicit incident facts.

Staff lens

— Design incident command across product, safety, security, privacy, legal, support, model, data, platform, and vendor owners so containment and communication authority is unambiguous.
— Use incidents to improve system boundaries and decision processes, not only prompts or model choice; require durable gates at the layer that allowed recurrence.

Interview defense

A support assistant gave harmful account guidance, the exact output is not reproducible, and the model team proposes a prompt patch. What do you do?

I would declare based on observed impact, not wait for deterministic reproduction. I would assign incident command and a harm owner, stop or narrow the affected capability through an independent control, preserve version and trace manifests, reconcile any account actions, and support affected users. I would test hypotheses across retrieval, policy, prompt, model, tools, orchestration, and product design; non-reproducibility is itself evidence about sampling and missing observability. A prompt patch can be one correction, but recovery needs a different bound revision, regression cases from the incident, deterministic boundary tests, canary guardrails, enough observation windows, and ready rollback. The safety case, evals, telemetry, runbooks, and ownership must change before closure where their assumptions failed.

Expect the interviewer to press on

— How do you preserve evidence without extending harmful exposure?
— When can a model behavior be a symptom rather than the root cause?
— What evidence is required before moving from containment to recovery?

Misconceptions to remove

“An AI incident cannot be confirmed unless the exact model output is reproducible.”

Credible user impact, effect receipts, screenshots, or trace evidence can justify declaration and containment; sampled outputs and changing dependencies often prevent exact replay.

“Fine-tuning or a stronger prompt closes a model incident.”

Visible output may arise from retrieval, authority, tooling, product, telemetry, or operational failures. Corrections and gates belong at every causal layer, with bounded recovery evidence.

“Blameless postmortems mean no individual or team owns the failure.”

Blamelessness supports honest learning; named owners still must implement controls, accept residual risk within authority, and verify corrective action.

Check your model

1. Why does containment precede root-cause certainty?

Ongoing exposure can create more harm while causal analysis is uncertain; deterministic controls can reduce impact without requiring a complete explanation.

2. What must a correction plan bind?

The incident and affected revision, root and contributing layers, owned actions, a different recovery revision, regression gates, canary evidence, rollback, and observation windows.

3. When is an incident ready to close?

After containment is stable, affected people and effects are addressed, recovery evidence and windows pass, communications are complete, residual risk is owned, and durable follow-ups have accountable tracking.

Prove the mechanism

Extend the playbook artifact with affected-cohort reconciliation and notification state. Make closure fail when either is incomplete, preserve exact types and content binding, and add deterministic recurrence ownership.

Add a production constraint

Run a tabletop for an agent that exposed cross-tenant data and may have changed tickets. Produce severity, command roles, kill actions, evidence handling, effect reconciliation, harmed-user support, layer hypotheses, correction gates, recovery canary, communication, safety-case changes, and closure criteria.

Artifact: AI incident response playbook

courses/ai-engineering/reference-impl/ai_incident_response/incident_playbook.py

Download reference implementation

Primary references and next links

References

1. Computer Security Incident Handling Guide (SP 800-61 Rev. 2)
National Institute of Standards and Technology. Primary incident-handling guidance on preparation, detection and analysis, containment, eradication, recovery, evidence, and post-incident learning.
2. AI RMF Playbook — Manage
NIST AI Resource Center. Official guidance on prioritizing, responding to, monitoring, and communicating AI risks and incidents.
3. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
National Institute of Standards and Technology. Primary profile on monitoring, incident disclosure, provenance, remediation, and feedback into generative-AI risk controls.

Continue through the graph

Observe the Decision Path, Not Just the Model Call →
Preserve versioned path evidence and expose gaps before an incident needs it.
Build a Safety Case for an AI Feature →
Reopen failed hazards, controls, assumptions, evidence, and launch conditions after an incident.

Glossary: incident declaration · containment · harm owner · evidence manifest · root cause · contributing factor · correction loop · regression gate · recovery canary · post-incident review