Arc 09 · Staff AI Engineer
Evaluation, Safety, and Reliability
Make quality a release system: task-specific datasets, calibrated judges, human review, security tests, telemetry, and incident learning.
Exit capability: Define a release bar that catches quality, safety, and reliability regressions.
- Mapped lessons
- 7
- Published now
- 7
- Full-arc estimate
- ≈17 hours
- Last edited
- 2026-09-22
Lesson sequence
Live units open into complete labs. Planned units stay visible to show the dependency path, but intentionally have no detail route. A withdrawn unit is retained only at its previously advertised URL and is not presented as live.
- 01
An Eval Is a Decision System
LiveStart from a release decision, then choose cases, graders, uncertainty, and escalation.
Design reviewAdvanced105 min estimateArtifact: Evaluation release decision contract
- 02
Build a Golden Dataset That Can Disagree with You
LiveSample real tasks, hard negatives, long tails, and policy boundaries without teaching to the test.
Build labAdvanced110 min estimateArtifact: Golden dataset coverage audit
- 03
Calibrate LLM Judges
LiveMeasure position bias, verbosity bias, self-preference, variance, and disagreement against human labels.
Failure labAdvanced115 min estimateArtifact: Calibrated judge agreement harness
- 04
Online Evaluation Without Shipping Blind
LiveConnect shadowing, canaries, holdbacks, user signals, and rollback thresholds.
Systems labAdvanced105 min estimateArtifact: Online evaluation rollout gate
- 05
Observe the Decision Path, Not Just the Model Call
LiveTrace retrieval, prompts, tools, model versions, policy checks, latency, and cost with controlled data capture.
Systems labAdvanced110 min estimateArtifact: Decision-path observability contract
- 06
Build a Safety Case for an AI Feature
LiveTie hazards to controls, evidence, residual risk, ownership, and launch conditions.
Design reviewAdvanced105 min estimateArtifact: AI feature safety case
- 07
AI Incident Response and Correction Loops
LiveContain harm, preserve evidence, distinguish model and system causes, and turn findings into durable gates.
Failure labAdvanced110 min estimateArtifact: AI incident response playbook