Systems labAdvanced105 min estimateOriginal publication

Online Evaluation Without Shipping Blind

Turn offline release evidence into a bounded production learning sequence whose assignment, guardrails, decision rights, and rollback path exist before exposure grows.

Authorship: InterviewsVector
Published / updated: 2026-09-22 / 2026-09-22
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary sources; they do not claim causal validity, production telemetry, safety certification, live incident response, or regulatory compliance.

The decision in one pass

Online evaluation is a sequence of decisions under exposure, not permission to replace offline evidence with production anecdotes. Start with a precise product decision and invariant assignment unit. Shadow the candidate where doing so is safe and representative, then canary it against a preserved control or holdback with stable eligibility, version, and analysis rules. Declare quality, safety, latency, cost, telemetry-completeness, sample, and observation-window gates before launch. Treat missing evidence as hold, a material regression as rollback, and passing one bounded stage only as permission to enter the next stage. Keep a kill path independent of the candidate, audit assignment and exclusions, examine heterogeneous harm rather than only averages, and separate causal experiment claims from operational canary comparisons.

Why this matters

Offline cases cannot reproduce every traffic shape, integration state, dependency failure, user adaptation, or distribution shift. Production signals are therefore necessary, but exposure creates real consequences and biased data. A team that increases traffic because aggregate engagement rose can hide safety regressions in a small cohort, compare non-equivalent populations, lose the control through retries or caching, or discover that its telemetry failed only after rollback evidence was needed.

You will be able to

Separate shadow, canary, randomized experiment, holdback, and full rollout by the decision each can support.
Define assignment, eligibility, exposure, interference, metric, and rollback contracts before production traffic is observed.
Combine quality, safety, reliability, cost, and telemetry-readiness signals without averaging away a hard guardrail.
Advance exposure in bounded stages and preserve an independently operable rollback path.
Communicate what an online result does and does not establish about causality and safety.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Map the release decision, assignment unit, candidate, control, eligibility, exposure path, metrics, guardrails, stop authority, and user-harm boundary.
02
Derive
Derive minimum evidence, observation windows, practical effect limits, segmentation, missing-data behavior, and stage-specific advance, hold, or rollback rules.
03
Build
Build a deterministic gate over invented aggregate candidate and holdback evidence; do not assign traffic or imply causal inference.
04
Stress
Inject incomplete telemetry, insufficient windows, cohort mismatch, safety events, quality loss, latency growth, cost growth, and rollback-path failure.
05
Operate
Monitor exposure, assignment integrity, guardrails, cohort balance, metric freshness, dependency health, decision latency, and rollback execution.
06
Defend
Defend why a passing canary authorizes only a larger bounded test and why a control population does not repair invalid assignment or missing outcomes.

Name the online decision before choosing the mechanism

Shadowing, canaries, experiments, and holdbacks answer different questions. Shadow traffic can expose integration and latency problems without showing candidate output to a user, but duplicate downstream effects must be disabled. A canary limits blast radius while observing real execution. A randomized controlled experiment can estimate a treatment effect only when assignment, exposure, outcomes, interference, and analysis are credible. A long-lived holdback estimates what would have happened without the new system, but it carries maintenance and fairness costs.

Mechanism	Best-supported decision	Critical boundary
shadow	is the candidate integration technically viable?	no user-visible or state-changing duplicate effect
canary	is bounded exposure within operational and harm limits?	fast independent rollback and cohort attribution
randomized experiment	did treatment cause a declared outcome change?	stable randomization, exposure logging, and analysis plan
holdback	what is the sustained incremental effect?	control preservation under product and model change

Bind assignment, evidence, and action in advance

01Freeze eligibility and assignmentChoose user, account, conversation, request, or another stable unit; persist treatment and version so retries, caches, and fallbacks cannot silently cross cohorts.
02Declare a metric contractName direction, denominator, window, practical limit, segmentation, late-event policy, missingness behavior, and source of truth for every metric.
03Separate guardrails from tradeable objectivesA safety ceiling, privacy breach, or rollback-path failure remains blocking even if engagement, quality, or cost improves elsewhere.
04Map each stage to one next actionAdvance, hold, rollback, or escalate must have named authority and executable mechanics; passing cannot mean automatic full release.

advance ⇔ readiness ∧ no_hard_regression ∧ exposure ≤ next_stage_limit

Readiness includes valid assignment, complete telemetry, the declared sample floor, and enough observation windows. A regression dominates readiness and triggers rollback rather than a larger sample.

Confidence intervals, power, sequential testing, and multiple-comparison control must match the inferential claim. The executable artifact below deliberately avoids those claims: it is a local operational gate over declared aggregate evidence, not an experiment-analysis library.

Audit one bounded rollout stage

online_rollout_gate.py

1def audit_rollout(contract: RolloutContract, evidence: RolloutEvidence) -> RolloutReport:
2    contract = validate_record(contract, RolloutContract)
3    evidence = validate_record(evidence, RolloutEvidence)
4    if evidence.scope != contract.scope or evidence.contract_content_id != contract.content_id:
5        raise ValueError("evidence belongs to another rollout contract")
6    # Readiness failures hold; declared metric regressions roll back.
7    # A clean stage can at most double exposure up to the contract cap.

Expected output

example=illustrative_only
decision=ADVANCE_CANARY
candidate_quality=0.940;control_quality=0.935
safety_rate=0.002;latency_ratio=1.080;cost_ratio=1.050
next_canary_fraction=0.100
claim=LOCAL_ROLLOUT_GATE_NOT_CAUSAL_PROOF

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/online_evaluation

The invented fixture compares 1,000 candidate observations with 1,000 control observations after three declared windows. It preserves shadow and holdback readiness, checks telemetry, quality, safety events, p95 latency, and mean cost, then moves a five-percent canary only to ten percent. The contract and evidence carry content digests and a shared eight-part scope; exact types, cohort roles, experiment identity, counts, bounds, constructor bypass, and digest tampering are tested.

The result does not prove randomization, representativeness, power, independence, correct logging, or safety. All cohorts and measurements are invented. `ADVANCE_CANARY` means only that this snapshot satisfies its declared local gate and may enter one more bounded observation stage.

Choose the next bounded rollout action

Inspect invented candidate and holdback evidence, predict advance, hold, or rollback, then compare your decision with the declared gate. No traffic is assigned and no production system is contacted.

Choose the next bounded rollout transition

Connect shadow evidence, canary stages, minimum observation windows, and rollback thresholds. Early good news never widens exposure by itself.

Rollout state

Shadow traffic exposes policy driftNo user sees the candidate, but offline-safe requests diverge in shadow.Canary crosses a safety rollback thresholdA small live cohort shows a declared high-severity regression.Healthy canary has not covered the windowEarly metrics pass, but the cohort has not crossed time and volume minima.Canary completes every stage gateQuality, safety, latency, volume, and rollback readiness remain healthy.

Traffic, regions, rates, and incidents are synthetic. The lab changes no deployment or live routing state.

Online gate evidence
Signal	Decision contract	Observed evidence
Exposure	Shadow receives no user-visible traffic	0% user-visible
Policy disagreement	Rate ≤ 0.5%	3.8%
Critical violations	Count = 0	7 synthetic boundary violations
Rollback need	No serving path changed	Not applicable; candidate is shadow-only

Synthetic evidence only. The decision remains hidden until you check a prediction.

What should the rollout controller do next?

Stay in shadow and repair the candidateRollback the canary immediatelyHold exposure and extend observationAdvance to the next bounded canary stage

Choose an outcome before checking the evidence contract.

Stress the evidence path, not only the candidate

Failure	Misleading observation	Required defense
sample-ratio mismatch	candidate appears unusually strong	audit assignment and exposure counts before reading outcomes
sticky-assignment break	one user experiences both systems	persist unit assignment across retries, channels, and caches
telemetry loss	safety rate appears to fall	hold on incomplete denominators or delayed harm events
novelty and learning	early engagement spike is treated as durable	use sufficient windows and a preserved holdback
average hides tail harm	global quality passes	segment by risk, tenant, language, task, and severity

Operate the decision log after the dashboard turns green

Record every stage with candidate and control revisions, assignment policy, eligibility, exposure, sample exclusions, telemetry completeness, raw metric definitions, uncertainty, segmented results, decision, approver, and rollback state. Keep the decision record immutable enough to reconstruct what evidence was visible when exposure changed. Re-evaluate after dependency, prompt, model, retrieval, policy, or product changes rather than inheriting a stale pass.

Alert on guardrail breaches and telemetry blindness separately; missing harm data is not a zero harm rate.
Track assignment drift, cross-cohort contamination, delayed outcomes, retry amplification, and fallback usage by declared unit.
Give support and incident responders the treatment, version, decision-log, and rollback identity needed to trace a report.
Retire experiment data under a declared privacy and retention policy; collect only signals that support a named decision.

Operate at three altitudes

Production lens

— Alert on hard safety, reliability, privacy, and telemetry guardrails independently of the optimization metric, with a tested route to reduce exposure immediately.
— Version assignment, candidate, control, metric definitions, analysis, and decision records so an operator can reconstruct exactly which system each unit experienced.
— Audit results by risk-bearing segments and delayed outcomes; an acceptable mean cannot authorize an unacceptable tail.

Staff lens

— Own online evaluation as a joint product, experimentation, platform, safety, privacy, support, and incident-response system rather than a model dashboard.
— Require every exposure stage to declare what new uncertainty it resolves, the maximum plausible harm, who can stop it, and what evidence permits the next stage.

Interview defense

An offline eval passes and a five-percent canary improves task completion, but safety-event telemetry is missing for one region. Do you advance?

No. Missing safety telemetry is a readiness failure, not evidence of zero events, so I would hold exposure at most and investigate the collection path. I would verify sticky assignment and version attribution, preserve the control, reconcile denominators and delayed events for the affected region, and check whether any irreversible harm requires rollback while evidence is restored. The predeclared gate should make that action deterministic. Only a complete window with hard guardrails passing can authorize the next bounded stage, and even then it does not prove safety or justify full rollout.

Expect the interviewer to press on

— When is a canary comparison not a causal experiment?
— How do retries and fallbacks contaminate treatment assignment?
— What information must be available to execute rollback during an incident?

Misconceptions to remove

“Production traffic is the most representative eval, so a small canary is always appropriate.”

Production can reveal realistic behavior, but some harms are unacceptable or irreversible even at small exposure and must be controlled before user testing.

“If the candidate beats the control on average, the rollout should advance.”

Hard safety and reliability guardrails, missing evidence, assignment invalidity, and harmed subgroups can block advancement regardless of an aggregate objective gain.

“Rollback means routing traffic back to the old model.”

A real rollback also covers prompts, retrieval indexes, policy, state, caches, data compatibility, tools, and operator authority; the old route must still be healthy and observable.

Check your model

1. Why does incomplete telemetry usually produce HOLD rather than PASS?

The denominator or adverse outcomes may be missing, so the gate lacks the evidence required to distinguish acceptable behavior from blindness.

2. What must be sticky in an online comparison?

The declared assignment unit and its candidate or control version across retries, caches, sessions, channels, and fallbacks, unless the design explicitly models crossover.

3. What does a passing canary establish?

Only that the observed bounded stage met its declared evidence and guardrail contract; it authorizes at most the next declared stage, not universal quality or safety.

Prove the mechanism

Add an assignment-integrity record and a segmented hard guardrail to the rollout artifact. Make telemetry loss produce HOLD, a harmful segment produce ROLLBACK, and preserve deterministic evidence IDs.

Add a production constraint

Design a sequential online evaluation for a multilingual support assistant. Specify unit, eligibility, interference, delayed outcomes, power or confidence method, multiple metrics, region-level safety ceilings, staged exposure, privacy, operator decision rights, and tested rollback.

Artifact: Online evaluation rollout gate

courses/ai-engineering/reference-impl/online_evaluation/online_rollout_gate.py

Download reference implementation

Primary references and next links

References

1. Canarying Releases
Google Site Reliability Engineering. Official operational guidance on canary populations, evaluation, staged rollout, and automated rollback decisions.
2. Online Experimentation at Microsoft
Microsoft Research. Primary report on operating controlled online experiments and using randomized evidence for product decisions.
3. AI RMF Playbook — Measure
NIST AI Resource Center. Official guidance on selecting risk measurements, acceptable limits, monitoring, and course correction.

Continue through the graph

An Eval Is a Decision System →
Carry a decision, evidence, uncertainty, and escalation contract into production exposure.
Observe the Decision Path, Not Just the Model Call →
Attribute online outcomes to the exact retrieval, policy, model, tool, and response path.

Glossary: shadow traffic · canary · holdback · assignment unit · guardrail metric · sample-ratio mismatch · interference · rollback threshold