Design reviewAdvanced105 min estimateOriginal publication

An Eval Is a Decision System

A useful evaluation does not end at a score. It binds a candidate, task distribution, grader policy, uncertainty method, release threshold, and accountable escalation path into one reviewable decision contract.

Authorship: InterviewsVector
Published / updated: 2026-09-22 / 2026-09-22
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary or official sources; they do not certify dataset quality, judge validity, product safety, or release fitness.

The decision in one pass

Define the decision before the metric. Name the candidate and baseline, the deployment task distribution, unacceptable failures, responsible decision owner, and possible outcomes: hold, collect evidence, escalate, or admit to release review. Then build cases that represent core work, important slices, long tails, adversarial conditions, and policy boundaries. Give each grader an explicit input, rubric, abstention path, calibration record, and authority limit. Bind every result to exact system, prompt, model, tool, dataset, rubric, grader, and policy versions. Report counts, slice results, regressions, uncertainty, and missing evidence rather than one average. Apply thresholds in a documented order so a critical failure cannot be hidden by a high aggregate. A passing offline gate is evidence for an accountable decision, not permission emitted by a benchmark and not proof of production safety.

Why this matters

Teams often collect a benchmark because it is available, optimize its mean, and decide what the number means after seeing it. That reverses the logic of measurement. It encourages test leakage, hides rare but costly failures, confuses grader preference with user value, and turns uncertainty into false precision. A decision system makes the release claim, evidence boundaries, and human ownership explicit before results create pressure to move the goalposts.

You will be able to

Translate a product release question into explicit hold, escalation, and review outcomes.
Bind cases, candidate outputs, baselines, graders, rubrics, and policies to immutable versions.
Design task and risk slices that prevent aggregate scores from hiding critical failures.
Use grader calibration, abstention, repeated evidence, and uncertainty in release decisions.
Operate evaluations as versioned production systems without claiming that an offline pass certifies safety.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Map the release owner, candidate, baseline, affected task distribution, harms, evidence producers, and reversible outcomes before selecting metrics.
02
Derive
Derive critical cases, slice support, regression limits, grader requirements, uncertainty rules, threshold order, and escalation ownership from the decision.
03
Build
Build content-bound evidence records and a deterministic gate that can hold, escalate, or admit a candidate to accountable review.
04
Stress
Inject critical misses, sparse slices, grader disagreement, abstentions, wide intervals, baseline regressions, stale identities, and constructor bypass.
05
Operate
Track evidence freshness, slice coverage, calibration drift, review queues, false-release and false-hold incidents, and online/offline disagreement.
06
Defend
Defend the exact claim supported by the gate and identify every material risk, population, or production condition it does not measure.

Write the decision before choosing the score

An evaluation is a designed information path from a question to an accountable action. Start with a decision such as whether candidate C may enter a limited canary for task population P under policy V. The evaluator must know what can happen next: block the release, request more evidence, escalate a judgment, or mark the candidate eligible for a separate release review. If the only output is 0.87, downstream teams will invent incompatible meanings for it.

Decision element	Contract question	Failure if omitted
scope	Which users, tasks, language, tools, and policy version?	results are generalized beyond evidence
candidate	Exactly which model, prompt, retrieval, tool, and config revisions?	a different system inherits the result
baseline	What current behavior must not regress?	a gain masks lost capability
loss	Which errors are costly or irreversible?	averages hide critical failures
action	Who may hold, escalate, canary, or release?	a metric silently becomes authority

Bind the full evaluated system, not a model name

For an AI product, behavior is produced by more than weights. Prompts, decoding, retrieval snapshots, feature flags, tool schemas, safety filters, memory, orchestration, and fallbacks all change the evaluated object. Record a content identity for the candidate and its dependencies, then attach every case outcome to one evaluation run. A result without those bindings is a story about an unknown system.

evaluation_scope = H(candidate, baseline, dataset, rubric, graders, interval_policy, release_policy)

The hash is an illustrative identity, not an authorization primitive. Any material component change invalidates reuse until policy explicitly says what evidence may carry forward.

01Freeze the targetRecord system and dependency revisions, runtime settings, tool grants, and data snapshot before generating outputs.
02Preserve raw evidenceKeep bounded inputs, outputs, grader versions, verdicts, rationales where appropriate, timestamps, and errors under governed access.
03Separate rerunsA changed candidate, grader, rubric, or dataset creates a new run rather than silently overwriting prior evidence.
04Compare like with likeEvaluate candidate and baseline on paired cases under equivalent budgets and environment when the decision depends on regression.

Connect cases, slices, graders, and failure costs

A case should exercise a behavior relevant to the release claim and carry enough provenance to explain why it exists. Organize cases into task and risk slices before aggregation: common requests, long tails, languages, customer tiers, tool paths, refusal boundaries, and known incident classes. Declare minimum support for each required slice. Eight copies of an easy core example do not cover a two-case policy boundary merely because the total is ten.

Grader	Good fit	Required control
deterministic check	schema, exact facts, constraints, tool traces	versioned oracle and error semantics
human review	contextual judgment and high-stakes ambiguity	rubric, training, disagreement, adjudication, privacy
model judge	bounded scalable comparison after validation	human calibration, swaps, repeats, abstention, drift checks
production signal	outcomes unavailable offline	causal caution, delayed labels, privacy, rollback

Make uncertainty and precedence part of policy

An observed rate is a sample statistic, not a permanent property. Report support and a prespecified uncertainty method, especially for small or high-variance slices. Do not choose the interval or threshold after seeing the candidate. A wide interval may justify collecting evidence rather than calling the system good or bad. Dependence, repeated templates, changing task mix, and noisy labels can make simple independent-case intervals optimistic, so disclose the sampling assumptions.

release evidence = point estimates + slice support + uncertainty + calibration + regressions + missingness

No single term substitutes for the others. A high mean with a critical miss, unsupported slice, invalid grader, or wide uncertainty should not pass through arithmetic averaging.

01Fail critical casesApply non-compensatory safety and policy conditions first.
02Validate evidenceCheck identity, support, grader calibration, abstention, and interval integrity.
03Check overall and slicesRequire both aggregate and declared segment thresholds.
04Check regressionsProtect baseline capabilities that the product still promises.
05Escalate uncertaintyRoute borderline or underspecified results to a named owner instead of rounding them into a pass.

Turn evidence into a release disposition

Inspect an illustrative candidate, baseline, task slices, critical cases, grader agreement, uncertainty, regressions, and abstentions. Predict whether fixed-order policy should hold, escalate, or admit the candidate to accountable release review. No model is called and no deployment is authorized.

Make the release decision from the contract

Start with the decision and inspect only evidence bound to its candidate, dataset, graders, slices, and uncertainty policy. Predict the valid outcome before revealing the gate result.

Evaluation record

Aggregate passes, boundary slice regressesThe overall score looks healthy while one policy slice falls below its gate.Point estimate passes, uncertainty is wideA small cohort clears the threshold, but the interval crosses the release bar.Scores use the wrong dataset revisionMetrics pass, but the evidence was computed against a superseded case set.Every declared gate is satisfiedIdentity, coverage, calibration, uncertainty, and critical slices all match.

All candidates, thresholds, cases, and scores are invented for practice. Eligibility still requires an accountable human release decision.

Release-decision evidence
Signal	Decision contract	Observed evidence
Overall pass rate	Lower bound ≥ 0.85	0.92; interval [0.88, 0.95]
Core slice	Lower bound ≥ 0.80	0.93; interval [0.88, 0.96]
Policy-boundary slice	Lower bound ≥ 0.80	0.79; interval [0.71, 0.85]
Critical cases	Every case passes	12 of 12 pass

Synthetic evidence only. The decision remains hidden until you check a prediction.

What does this evidence authorize?

Eligible for accountable release reviewHold the candidate releaseEscalate and collect stronger evidenceReject the evidence as out of contract

Choose an outcome before checking the evidence contract.

Audit a release contract without releasing anything

eval_release_gate.py

1def evaluate_release(contract, evidence):
2    """Apply a fixed-order release gate to scope-bound evaluation evidence."""
3    contract = validate_record(contract, EvaluationContract)
4    evidence = validate_record(evidence, EvaluationEvidence)
5    if evidence.scope != contract.scope:
6        raise ValueError("evidence scope does not match the release contract")
7    if evidence.contract_content_id != contract.content_id:
8        raise ValueError("evidence belongs to another release contract")

Expected output

example=illustrative_only
decision=ELIGIBLE_FOR_RELEASE_REVIEW
first_reason=none;action=accountable-human-review
cases=8;passes=8;abstentions=0;regressions=0
pass_rate=1.000;interval=[0.880,1.000]
claim=LOCAL_EVAL_GATE_NOT_PRODUCTION_CERTIFICATION

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/eval_system

The invented fixture binds eight outcomes to one contract, run, baseline, required slice set, allowed grader set, uncertainty method, and calibration declaration. The policy evaluates critical failures, calibration support, grader agreement, interval width, abstentions, lower-bound quality, slice quality, and baseline regressions in a stable order. `ELIGIBLE_FOR_RELEASE_REVIEW` deliberately stops short of `RELEASED`.

The artifact reconstructs frozen records, copies caller-owned collections, checks content digests and scope, rejects duplicate or undersupported evidence, exact-type violations, non-finite ratios, string subclasses, tampering, and constructor bypass. Its interval, outcomes, thresholds, agreement, and calibration support are declared fixtures. It does not recompute a statistical interval, call a grader, authenticate a reviewer, inspect production, or certify the release.

Operate the measurement system and its blind spots

Operational signal	Question	Response
coverage drift	Does traffic leave the represented task and risk slices?	sample new cases and disclose unsupported regions
grader drift	Does agreement change by rubric, slice, or version?	recalibrate, route disagreement, or retire the grader
offline/online gap	Do production outcomes contradict the gate?	hold rollout and investigate construct or sampling failure
evidence age	Are labels, policies, corpora, and tools still current?	expire or revalidate evidence
gate pressure	Are teams tuning only to known cases?	protect holdouts, rotate probes, and audit leakage

Store evaluation manifests and decision records with access controls and retention appropriate to their data. Monitor the evaluator as a production dependency: queue delay, error rate, grader cost, abstention, reviewer backlog, reproducibility, and decision overrides. When a production incident escapes, add a durable regression case only after identifying the failure mechanism; copying the symptom without coverage analysis can create a brittle patchwork benchmark.

Operate at three altitudes

Production lens

— Version candidate, baseline, dataset, rubric, grader, uncertainty method, release policy, and decision evidence under one immutable evaluation identity.
— Alert on missing required slices, stale evidence, grader disagreement, abstention spikes, wide uncertainty, critical failures, baseline regressions, and review backlog.
— Separate offline eligibility, canary authorization, rollout progression, and final release; each boundary needs its own owner and rollback condition.

Staff lens

— Make release criteria a cross-functional risk contract owned by product, domain, safety, security, and operations—not a private benchmark owned only by model developers.
— Fund evaluation infrastructure as a governed evidence system with lineage, access control, reproducibility, reviewer capacity, incident feedback, and explicit unsupported claims.

Interview defense

Design an evaluation system for deciding whether a new AI support assistant can enter production.

I would first specify the decision, population, current baseline, unacceptable failures, accountable owner, and outcomes such as hold, gather evidence, escalate, or enter canary review. I would content-bind the whole candidate system and build a versioned case set from real tasks plus long-tail, policy, safety, and tool-use slices with minimum support. I would use deterministic graders where possible, calibrated human or model judgment where necessary, and an abstention path. The scorecard would report paired regressions, critical cases, per-slice results, support, disagreement, and uncertainty under a prespecified method. A fixed-order gate would fail critical conditions before aggregates and would escalate weak calibration or wide uncertainty. I would preserve raw evidence, monitor coverage and grader drift, connect offline results to canary outcomes, and state that passing admits accountable review rather than certifying safety.

Expect the interviewer to press on

— How would you prevent a high aggregate from hiding a critical policy failure?
— When should the gate collect more evidence rather than block the candidate?
— What evidence becomes stale when only the prompt changes?

Misconceptions to remove

“An eval is a dataset plus an average score.”

A release eval also needs a scoped candidate, baseline, graders, uncertainty, threshold precedence, missing-evidence rules, and an accountable action path.

“Passing the offline gate proves the system is safe to release.”

It supports only the declared decision under its evidence and assumptions. Production safety also depends on controls, deployment match, canaries, monitoring, rollback, and accountable residual-risk acceptance.

“More cases always make the evaluation more valid.”

Repeated easy or correlated cases can increase count without improving construct, population, or risk coverage. Coverage design and provenance matter.

Check your model

1. Why should a critical-case rule precede an aggregate quality threshold?

Because the release contract declares that some failures are non-compensatory. Allowing many easy passes to average away a critical failure would contradict the decision policy.

2. What does a wide uncertainty interval imply?

Usually that the evidence cannot support a precise release claim under the chosen sampling assumptions. The correct action may be targeted collection or escalation, not optimistic rounding.

3. Why bind grader and rubric versions to the run?

A changed rubric or grader can change verdict semantics. Without version binding, apparent candidate movement may be measurement movement.

Prove the mechanism

Extend the illustrative release contract with one declared tool-trajectory slice and a deterministic trace grader. Add a fixed-order rule for missing tool evidence, tests for candidate/baseline pairing, and a decision-record field that names the accountable reviewer without turning that identity into an authorization claim.

Add a production constraint

Design a sequential evidence-collection policy for a costly human-reviewed slice. State its stopping and maximum-sample rules in advance, simulate both clear and borderline candidates, and explain how repeated looks, dependence, and reviewer drift affect the claimed uncertainty.

Artifact: Evaluation release decision contract

courses/ai-engineering/reference-impl/eval_system/eval_release_gate.py

Download reference implementation

Primary references and next links

References

1. OpenAI evaluation best practices
OpenAI. Official guidance on task-specific objectives, representative datasets, explicit metrics, and continuous evaluation.
2. NIST AI Risk Management Framework 1.0
NIST. Primary framework for governed measurement, risk management, documentation, and accountability.
3. NIST AI 600-1 Generative AI Profile
NIST. Primary profile covering empirical evaluation of generative-AI capabilities, limitations, and risk controls.
4. Confidence intervals for a proportion
NIST/SEMATECH. Official statistical reference for expressing uncertainty around a measured proportion.

Continue through the graph

Build a Golden Dataset That Can Disagree with You →
Build the governed task and risk evidence consumed by a release evaluation.
Calibrate LLM Judges →
Validate a scalable grader before its verdicts influence release policy.

Glossary: decision contract · construct validity · critical case · slice support · grader calibration · abstention · regression gate · uncertainty interval