Concept labFoundation65 min estimateOriginal publication

Define the Measurement Before the Model

A model score cannot tell a team what to ship. A measurement contract connects a product claim to a population, evidence, guardrails, and a decision made before results are visible.

Authorship: InterviewsVector
Published / updated: 2026-08-11 / 2026-08-11
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Define success before selecting a model: write the product claim, eligible population, decision owner, primary user outcome, offline evidence, online evidence, safety guardrails, latency objective, and cost boundary. Give every measure a unit, direction, threshold, slice, window, data source, denominator, and action on failure. Bind each observation to the exact definition and contract digests plus a versioned evidence source, provenance identifier, and verified evidence-manifest digest. Offline quality is evidence that an implementation might work; it is not proof that the product claim moved. A release ships only when scoped evidence is complete and every blocking gate passes.

Why this matters

Without a precommitted measurement contract, teams optimize the easiest available score, reinterpret results after seeing them, and discover safety, latency, or unit-cost regressions only after rollout. The model becomes the project even when the user outcome never improves.

You will be able to

Turn an ambiguous product claim into a falsifiable decision contract.
Separate offline capability evidence from causal online product evidence.
Specify safety, latency, and cost as blocking constraints rather than dashboard decoration.
Detect missing telemetry, invalid denominators, slice regressions, and post-result threshold changes.
Produce a deterministic release decision only from scoped, versioned, provenance-bound observations.

Prerequisite contract

— Ratios, rates, and percentiles
— Basic experimental reasoning
— A written product or workflow claim

Your Vector Loop for this lab

01
Model
Represent measurement as a contract between a product claim and a release decision.
02
Derive
Derive each metric from a named population, event, denominator, direction, and threshold.
03
Build
Implement typed gates that reject bare values and mismatched scope, definition, contract, or evidence provenance.
04
Stress
Break the contract with proxy gaming, missing data, slice failures, and invalid comparisons.
05
Operate
Version metric definitions, telemetry, windows, and decision evidence with each release.
06
Defend
Explain why model quality is necessary evidence but not sufficient authority to ship.

Start with the decision the evidence must support

A useful product claim names a person, a task, a condition, and a desired change. “Use AI to improve support” is not measurable. “For eligible support cases, help an agent resolve a reviewed case correctly with less effort while preserving escalation and privacy constraints” creates a boundary. It identifies an outcome, an evaluation unit, exclusions, and harms that must not be traded away.

The contract is complete only when evidence can trigger a concrete action.
Contract question	Required answer	Failure if omitted
What decision?	ship, hold, expand, roll back, or investigate	a dashboard with no consequence
For whom?	eligible population plus important slices	aggregate gains hide harmed groups
Compared with what?	baseline or concurrent control	movement has no counterfactual
Over what window?	observation and maturation period	early or delayed outcomes are misread
Who decides?	named accountable role	metric conflict becomes an ownership dispute

Separate five evidence domains

Domain	Question it answers	Example evidence	What it cannot prove
Offline	Can this implementation perform the defined task on reviewed cases?	task score by difficulty slice	causal product value
Online	Does exposure change the intended user or business outcome?	controlled completion-rate difference	long-term safety by itself
Safety	Are forbidden or harmful outcomes within tolerance?	unsafe-action rate and severity	overall usefulness
Latency	Can users receive a usable result within the service objective?	end-to-end p95 by request class	semantic correctness
Cost	Is the delivered outcome inside its economic boundary?	cost per reviewed completion	value without the outcome denominator

rate = count(events satisfying condition) / count(eligible opportunities)

Name both sets. Changing eligibility, losing events, or allowing the treatment to change the denominator can move a rate without improving the underlying outcome.

Direction and threshold belong to the metric definition. “Higher task score is better” is incomplete without the minimum acceptable value, protected slices, uncertainty rule, and action when data is insufficient. Decide those rules before observing the candidate result.

Build a gate that fails closed on missing evidence

measurement_contract.py

1ILLUSTRATIVE_VALUES = (0.84, 1.3, 0.008, 940.0, 0.054)
2ILLUSTRATIVE_OBSERVATIONS = tuple(
3    _illustrative_observation(
4        gate,
5        value,
6        f"illustrative-evidence-payload:{gate.name}:{value}",
7    )
8    for gate, value in zip(
9        ILLUSTRATIVE_CONTRACT.gates,
10        ILLUSTRATIVE_VALUES,
11    )
12)
13 
14if __name__ == "__main__":
15    print(format_example())

Expected output

example=illustrative_only
contract=measurement-contract-v1
offline:offline_task_score=PASS (0.840 >= 0.820) evidence=illustrative_offline_evidence_v1
online:online_completion_delta=PASS (1.300 >= 1.000) evidence=illustrative_online_evidence_v1
safety:unsafe_action_rate=PASS (0.008 <= 0.010) evidence=illustrative_safety_evidence_v1
latency:p95_latency_ms=FAIL (940.000 <= 900.000) evidence=illustrative_latency_evidence_v1
cost:cost_per_completion_usd=PASS (0.054 <= 0.060) evidence=illustrative_cost_evidence_v1
decision=HOLD
failure_actions=hold_release_and_investigate_latency

Verify: Run python3 -m unittest discover courses/ai-engineering/reference-impl/measurement_contract.

The example is deliberately a hold even though four gates pass. Each MetricObservation copies the promised slice, window, data source, denominator, failure action, definition version and digest, contract version and digest, evidence version and provenance, and a manifest whose digest is recomputed at construction. Evaluation rejects bare name-to-float maps or any mismatched binding before comparing values. The numbers and evidence references are invented fixtures used only to demonstrate mechanics; a real contract must derive gates from product requirements, baseline evidence, risk tolerance, and service economics.

Attack the measurement, not only the model

01Move the denominatorExclude hard requests after treatment or count only completed responses. Report numerator and denominator separately and freeze eligibility before assignment.
02Average away a harmed sliceAn aggregate can improve while a language, tenant, accessibility, or risk slice regresses. Predeclare protected slices and minimum sample handling.
03Tune on the evaluation setRepeated prompt or model selection against the same cases converts the test set into training feedback. Keep a final holdout and version case exposure.
04Change telemetry in one variantA logging change can look like behavior change. Gate on event completeness, assignment integrity, and schema parity before interpreting outcome movement.
05Select the threshold after resultsPost-result rules convert evidence into a negotiation. Record the decision rule before candidate evaluation and log exceptions explicitly.

Carry the contract through release and production

Release evidence	Version with	Operational use
case set and labels	dataset and rubric	reproduce offline claims
metric implementation	query/code and schema	detect definition drift
online assignment	experiment and exposure IDs	preserve causal analysis
service indicators	route, client, and request class	diagnose tail latency and errors
decision record	candidate and contract version	audit ship, hold, or exception

After launch, monitor the same user outcome and guardrails with production-appropriate windows. Add drift and incident signals, but do not let a large monitoring catalogue replace the few measures that govern the product decision. Reopen the contract when the population, workflow, harm model, price basis, or service promise changes.

Operate at three altitudes

Production lens

— Emit numerator, denominator, eligibility version, slice, metric definition digest, contract digest, and evidence provenance together.
— Monitor assignment integrity and event completeness before interpreting product metrics.
— Keep client-observed latency and failure rates alongside model-server measurements.
— Store release decisions and exceptions with the exact contract and candidate versions.

Staff lens

— Make metric ownership cross-functional: product owns the claim, domain experts own validity, engineering owns reliable collection, and risk owners set non-tradable constraints.
— Ask which decision would change if a metric moved; remove vanity measures that cannot change action.
— Separate recommendation from evidence when threshold selection depends on organizational risk tolerance.

Interview defense

How would you measure whether a new AI feature should ship?

I would first define the user claim, eligible population, baseline, decision owner, and release action. Then I would predeclare offline task evidence, a causal online outcome where feasible, safety guardrails, end-to-end latency/reliability objectives, and cost per useful outcome. Every metric needs a denominator, slice, window, threshold, data source, failure action, definition version, and evidence-provenance rule. Observations must bind to the exact contract and definition digests; a bare value cannot authorize release. I would validate telemetry and assignment, compare the candidate within the authorized scope, and hold if any blocking gate or required evidence fails.

Expect the interviewer to press on

— When is an online experiment infeasible or unethical?
— How do you handle a global win with a slice regression?
— Why is model accuracy not the primary product metric?

Misconceptions to remove

“A statistically significant offline improvement means the feature should ship.”

Offline movement can support capability evidence but cannot establish causal product value or satisfy safety, latency, and cost constraints.

“More metrics make a decision more rigorous.”

Unowned or redundant metrics create interpretation flexibility. A small predeclared decision set plus diagnostic measures is more defensible.

“No observed safety event means the safety rate is zero.”

It may mean insufficient exposure, missing detection, or a rare event. Report observation volume, detection coverage, and uncertainty.

Check your model

1. Why must a rate specify its denominator and eligibility rule?

Because treatment, logging, or filtering can change which opportunities are counted; the rate can move even when the underlying outcome does not.

2. What should happen when a blocking safety observation is missing?

The release should hold or enter a predeclared limited mode. Unknown evidence must not be interpreted as a passing zero.

Prove the mechanism

Choose one AI product claim and write a measurement contract with one primary outcome, at least one offline measure, one online measure or justified alternative, one safety guardrail, one latency objective, and one cost boundary. Define every denominator and failure action.

Add a production constraint

Add slice-specific gates, an inconclusive state for insufficient evidence, and a signed exception record without allowing an exception to rewrite the original result.

Artifact: Measurement contract

courses/ai-engineering/reference-impl/measurement_contract/measurement_contract.py

Download reference implementation

Primary references and next links

References

1. Artificial Intelligence Risk Management Framework 1.0
National Institute of Standards and Technology. Primary government framework for mapping, measuring, and managing AI risks and deployment-relevant evidence.
2. Service Level Objectives
Google Site Reliability Engineering. Official guidance for defining user-relevant indicators, objectives, windows, and consequences.
3. Online Experimentation at Microsoft
Microsoft Research. Primary research on controlled experiments for evaluating causal product impact.
4. The ML Test Score
Breck et al.. Primary paper on production-readiness tests and monitoring beyond model accuracy.

Continue through the graph

Diagnose RAG by Stage →
Apply the measurement discipline to a multi-stage AI system.
Engineering baseline roadmap →
Place measurement beside reproducibility, data contracts, and service boundaries.

Glossary: product claim · guardrail metric · denominator · controlled experiment · SLI · SLO · decision gate