Build a Safety Case for an AI Feature
Replace the claim that a feature is safe with a reviewable argument: bounded claims, named hazards, independent evidence, residual-risk decisions, owners, and conditions that remain true after launch.
- Authorship
- InterviewsVector
- Published / updated
- 2026-09-22 / 2026-09-22
- Review status
- Artifact tests passing · primary sources recorded
Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary sources; they do not claim causal validity, production telemetry, safety certification, live incident response, or regulatory compliance.
The decision in one pass
A safety case is a structured, challengeable argument for a bounded feature in a named operating context—not a certificate that an AI system is safe. Define the claim and exclusions first. Identify plausible harms, exposed people, pathways, severity, likelihood, uncertainty, and misuse. For every material hazard, name preventive, detective, responsive, and recovery controls; link each control to versioned evidence with an owner and freshness requirement. Estimate residual risk without laundering uncertainty into a precise score, record who accepts it and under what authority, and make launch conditions observable and revocable. Use independent review for material claims, preserve dissent and gaps, connect assumptions to monitoring and incident thresholds, and reopen the case whenever the model, data, retrieval, tools, policy, user population, or environment changes.
Why this matters
A launch checklist can show that teams completed tasks while leaving the central argument implicit: which harms matter, why the controls should prevent or limit them, what evidence supports that belief, which uncertainty remains, and who is accountable if an assumption fails. AI features change through model updates, distribution shift, prompt and retrieval changes, tool permissions, and user adaptation, so a one-time review can become false assurance quickly.
You will be able to
- Write a bounded safety claim with an explicit operating context, population, exclusions, and expiry triggers.
- Connect hazards to prevention, detection, response, and recovery controls without confusing controls with evidence.
- Assess residual risk and uncertainty after controls, then name an accountable acceptance authority and launch conditions.
- Use independent challenge, traceable evidence, monitoring, and incident feedback to keep the case alive.
- Recognize when missing evidence or unacceptable risk requires blocking launch rather than adding caveats.
Your Vector Loop for this lab
- 01
Model
Map the feature boundary, intended and foreseeable use, people exposed, harms, system dependencies, authority, assumptions, and lifecycle changes.
- 02
Derive
Derive explicit hazard claims, inherent and residual risk, uncertainty, control obligations, evidence standards, launch conditions, and review triggers.
- 03
Build
Build a deterministic audit of an invented hazard-control-evidence graph; do not certify a system or invent evidence for a real launch.
- 04
Stress
Remove or expire evidence, mismatch controls, increase residual risk, bypass review, change scope, and add plausible misuse or distribution shift.
- 05
Operate
Monitor hazard indicators, control health, evidence freshness, assumptions, exceptions, user reports, near misses, incidents, and ownership after launch.
- 06
Defend
Defend the claim boundary, remaining uncertainty, why each evidence item supports its control, and who can block, accept, or revoke launch.
Bound the claim before collecting reassuring evidence
Write the claim so another reviewer can falsify it. Name the feature revision, users, tasks, geography, data, integrations, autonomy, impact, operating limits, and excluded uses. A statement such as ‘the assistant is safe’ has no useful boundary. A better claim identifies a specific harm and condition, such as preventing unsupported account actions from reaching users without a verified source or human escalation under the released support workflow.
- 01Describe exposed people and assetsInclude direct users, people represented in data, operators, bystanders, downstream recipients, organizations, and environmental or societal interests when relevant.
- 02Trace harm pathwaysConnect initiating conditions, model or system behavior, failed boundaries, exposure, and consequence instead of naming only a generic risk category.
- 03Record assumptions and uncertaintyMark where data is sparse, likelihood is subjective, severity varies by population, or evidence comes from a proxy environment.
- 04Name invalidation triggersModel, prompt, retrieval, data, policy, tool, population, scale, regulation, or incident changes can require re-review.
Connect each control to independent evidence
| Layer | Example obligation | Evidence that could support it |
|---|---|---|
| prevent | restricted action cannot bypass authorization | negative integration tests plus policy-service audit |
| detect | unsupported guidance is identified before delivery | labeled boundary cases and monitor validation |
| respond | operator can disable the affected path | timed incident drill with independent observation |
| recover | affected outcomes can be found and corrected | reconciliation exercise and user-remediation procedure |
A control is a mechanism or process; evidence is the observation that supports a claim about it. ‘We have a filter’ is not evidence that the filter covers a hazard, works at the deployed version, resists bypass, stays observable, and has an owner. Prefer heterogeneous evidence: design analysis, unit and integration tests, adversarial tests, human studies where appropriate, operational drills, monitoring validation, historical incidents, and independent review.
residual_risk = risk after credited controls, conditioned on evidence, uncertainty, and operating assumptions
Do not subtract a control because it exists on a diagram. Credit only the bounded effect supported by current evidence, and preserve uncertainty separately from the score.
Audit the hazard-control-evidence graph
1def audit_safety_case(contract: SafetyCaseContract, case: SafetyCase) -> SafetyCaseReport:2 contract = validate_record(contract, SafetyCaseContract)3 case = validate_record(case, SafetyCase)4 if case.scope != contract.scope or case.contract_content_id != contract.content_id:5 raise ValueError("case belongs to another safety contract")6 # Every control must have linked passing evidence and residual risk must meet the bound.7 # Independent review is a gate, not a decorative field.Expected output
example=illustrative_only
decision=READY_WITH_CONDITIONS
maximum_residual_risk=6
hazard_conditions=1
violations=0
claim=LOCAL_SAFETY_CASE_AUDIT_NOT_SAFETY_CERTIFICATIONVerify: python3 -m unittest discover courses/ai-engineering/reference-impl/safety_case
The invented fixture describes one unsupported-guidance hazard, two named controls, a test record, a process drill, a residual score, a feature owner, an independent reviewer, and a launch condition. The audit rejects missing, failed, expired, mismatched, unlinked, or duplicate evidence; unsubstantiated controls; excessive residual risk; incomplete review; changed scope; constructor bypass; non-exact types; and content-digest tampering.
The audit checks internal completeness, not real safety. It does not discover hazards, run tests, authenticate an evidence owner, validate numeric risk, assess legal requirements, approve launch, or certify compliance. All records are invented, and `READY_WITH_CONDITIONS` retains rather than erases the condition attached to each hazard.
Make disagreement, acceptance, and stop authority explicit
Material safety claims need review with enough independence, expertise, time, and access to challenge the feature team. Record unresolved objections and evidence gaps. A business owner may accept some residual product risk but cannot waive another team's legal, security, privacy, or safety authority. The decision record must show which risks are accepted, transferred, mitigated, avoided, or deferred—and which conditions immediately revoke launch.
| Decision state | Meaning | Required next step |
|---|---|---|
| block | risk exceeds authority or evidence is missing | do not expose; redesign or produce valid evidence |
| ready with conditions | bounded residual risk accepted under live constraints | launch only with monitoring, owner, expiry, and kill path |
| exception | temporary departure from a control | name compensating controls, approver, scope, and expiry |
| reopen | assumption, revision, environment, or incident changed | invalidate affected claims and repeat review |
Turn incidents and drift into revised claims
Attach every evidence item to the tested revision, environment, population, date, owner, and expiry. Monitor leading indicators, near misses, support reports, exceptions, control availability, and actual incidents. When one occurs, preserve the original case and decision, identify which hazard, control, evidence, or assumption failed, repair the system, add a regression gate, and issue a new version. Learning should narrow uncertainty or reveal a new boundary, not rewrite the old record to look prescient.
- Review control health and evidence freshness on a risk-based cadence, not only before launch.
- Track who owns each hazard and condition when teams reorganize or vendors change.
- Sample real use for previously unidentified pathways while respecting privacy and acceptable-exposure limits.
- Publish a concise residual-risk and limitation statement to operators and decision makers who rely on the feature.
Operate at three altitudes
Production lens
- — Monitor every launch condition, credited control, evidence-expiry date, hazard indicator, exception, near miss, and incident with a named owner and revocation action.
- — Bind safety claims and evidence to exact model, data, retrieval, prompt, tool, policy, and product revisions; invalidate only affected claims through an explicit review.
- — Exercise containment, rollback, escalation, and recovery controls rather than treating their documented existence as operating evidence.
Staff lens
- — Create decision rights that let independent safety, security, privacy, legal, product, and operations owners challenge evidence and block within their authority without diffusing final accountability.
- — Keep the case legible to executives and implementers: a traceable argument with uncertainty and conditions, not a warehouse of disconnected test reports.
Interview defense
A feature team presents a high pass rate and asks for safety approval. How do you turn that into a launch decision?
I would first bound the feature, population, tasks, integrations, and excluded uses, then identify plausible harms and their pathways. I would map each material hazard to preventive, detective, responsive, and recovery controls, and ask for versioned evidence that actually supports each control under representative and adversarial conditions. I would record uncertainty and residual risk rather than treating the pass rate as a universal score. Independent reviewers and the appropriate risk owners must resolve or preserve dissent, and launch would remain conditional on observable controls, owners, expiry triggers, incident thresholds, and a tested stop path. Missing evidence or risk above authority means block, not approve with vague monitoring.
Expect the interviewer to press on
- — What is the difference between a control and evidence?
- — Who can accept residual risk when several governance domains overlap?
- — Which changes should reopen the case after launch?
Misconceptions to remove
“A safety case proves the AI feature is safe.”
It makes a bounded argument and its evidence reviewable. Unknown hazards, invalid assumptions, drift, and evidence error remain; the case must state limitations and change triggers.
“A passing test is a control.”
A control changes risk; a test is evidence about a control under specific conditions. Passing stale or unrepresentative tests cannot keep the control effective.
“Residual risk can be summarized by one precise number.”
Scores can prioritize, but scenario differences, uncertainty, affected populations, irreversible outcomes, and authority boundaries must remain visible.
Check your model
1. What makes a safety claim falsifiable?
It names the feature revision, context, population, harm, operating conditions, measurable boundary, and evidence that would refute or invalidate it.
2. Why must evidence be bound to a revision and expiry?
A result supports only the tested system and context; model, data, tools, policy, users, and environments change, while monitors and drills can become stale.
3. What should happen when a launch condition stops holding?
The named response—reduce exposure, disable capability, escalate, or block—should execute, and affected claims should reopen with the event preserved as new evidence.
Prove the mechanism
Add a second hazard and a freshness field to the safety-case artifact. Make expired evidence block only the claims that rely on it, preserve independent review, and emit deterministic condition ownership.
Add a production constraint
Write and defend a safety case for a tool-using support assistant. Include indirect injection, incorrect account action, privacy leakage, over-reliance, disparate failure, abuse, operator error, vendor dependency, controls, evidence, residual risk, decision rights, launch conditions, monitoring, and reopen triggers.
Artifact: AI feature safety case
courses/ai-engineering/reference-impl/safety_case/safety_case_audit.py
Download reference implementationPrimary references and next links
References
- 1. Artificial Intelligence Risk Management Framework (AI RMF 1.0)
National Institute of Standards and Technology. Primary framework for governing, mapping, measuring, and managing AI risks across the lifecycle.
- 2. Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile
National Institute of Standards and Technology. Primary profile on generative-AI risks, measurement, monitoring, incident disclosure, and residual-risk responses.
- 3. CDEI Portfolio of AI Assurance Techniques
UK Centre for Data Ethics and Innovation. Official portfolio describing how assurance techniques can support justified claims about AI-system trustworthiness.
Continue through the graph
- Agent Red-Team: Untrusted Instructions Meet Real Tools →
Use adversarial system evidence to support, challenge, or narrow specific safety claims.
- AI Incident Response and Correction Loops →
Feed failed assumptions and controls back into a versioned living safety case.
Glossary: safety case · hazard · harm pathway · inherent risk · residual risk · control · assurance evidence · independent review · launch condition