Lesson 10 of 23·Deployment Patterns for ML: Why Blue/Green Fails for Models

Module 3 · Lesson 3 · Core · 32 min

Deployment Patterns for ML: Why Blue/Green Fails for Models

Blue/green is a deploy pattern for code, where the failure mode is crashes. Models need a different one because their failure mode is silent quality drift. This lesson covers the shadow → canary → interleaved/A/B → ramped pattern and when each step is the right call.

Framework: The Regression Surface

The deploy patterns engineering teams have spent two decades refining for code — blue/green, canary, feature flags — were designed around a specific failure mode: the new code crashes or returns errors, and you roll back. ML changes don't fail that way. ML changes fail by being subtly worse on a slice of traffic that nobody notices for three weeks, by the time the team has shipped two more model versions on top. Applying blue/green to ML is like applying a fire alarm to flood detection — the right alarm for the wrong failure.

The Quality-Aware Rollout Ladder is the four-step progression designed around the silent-quality-drift failure mode. Shadow, canary, interleaving or A/B, ramped with holdback. Each step catches a class of failure that the previous step couldn't, and the order matters — shadow catches gross failures before any user sees them, canary catches quality regressions on small traffic, interleaving or A/B measures whether the change is actually better, and the permanent holdback measures whether the improvement persists at retention timescale. Skipping a step is rarely catastrophic, but compounding skipped steps is how ML platforms drift into a state where no one trusts the model changes anyone ships.

Framework

The Regression Surface

Shadow, canary, A/B, and ramped aren't a sequence — they're a coverage map over a five-dimensional regression surface, and which dimension a rollout step actually covers is what determines whether you need that step for a given change. The Staff move is to refuse the 'always run shadow then canary' default and instead name which dimensions the change touches, then run only the steps that cover those dimensions. Skipping a step doesn't just save time; it leaves dimensions uncovered, and the cost of leaving them uncovered is the cost of the regression class they catch.

1
Dimension 1 — Data shift
The input distribution changed (upstream schema, new content class, seasonal). Coverage: always-on Layer 1 observability + shadow mode. Canary and A/B alone won't catch a data shift that affects all variants equally. The under-monitored dimension; most teams catch this only after Tier 3 accuracy degrades.
2
Dimension 2 — Model shift
The model's outputs structurally differ — calibration drift, output distribution change, slice-specific behavior. Coverage: shadow (if output distribution monitoring exists), canary (if per-class quality monitoring exists). Aggregate accuracy hides this because the shift is usually slice-specific.
3
Dimension 3 — Integration shift
Downstream consumers see a different contract — output schema, latency profile, caching behavior. Coverage: shadow catches most, canary catches integration-load effects. The dimension most often missed during 'we just changed the model' deploys; the source of the 'right model, broken downstream' incident class.
4
Dimension 4 — Performance shift
Latency, throughput, cost-per-request. Coverage: shadow at production-equivalent load, canary at real production load. Quantization, model-size changes, and serving-infra updates land here. A/B is the wrong instrument because variance in serving latency dominates the A/B signal at small traffic.
5
Dimension 5 — Business shift
The system functions but the business metric moved the wrong way. Coverage: only A/B and long-term holdback. This is the most expensive dimension because it's the slowest to detect — retention-timescale effects take months and require a permanent holdback group. The dimension every team underinvests in because the operational cost of maintaining the holdback is real.
6
How to apply: name the dimensions, then pick the steps
Every change touches a subset of the five dimensions. A config-only change: dimension 4 maybe, 1-3 and 5 untouched — skip shadow, canary only. A new feature in the pipeline: 1, 2, 3 — full shadow and canary mandatory, A/B for the 5 confirmation. A new model architecture: all 5 — full ladder with extended A/B and permanent holdback. The default 'always run shadow then canary then A/B' wastes engineering time on dimensions the change doesn't touch and under-invests on the ones it does.

When to use

Apply the Regression Surface to any model deployment and to any review of an existing rollout policy. The diagnostic question is 'which dimensions does this change touch?' — the answer determines which rollout steps are mandatory and which are optional. Teams that run the full ladder on every change waste engineering time; teams that run only canary skip the dimensions only shadow catches. The right answer is dimension-aware step selection, not a fixed sequence.

Worked example

Senior: 'Always run shadow then canary then A/B.' Staff: 'This change is a quantization update — it touches dimensions 2 and 4 but not 1, 3, or 5. So mandatory: shadow with output distribution monitoring and load-equivalent latency comparison. Optional: canary if we want production-load confirmation. Not needed: A/B against business metric — quantization doesn't change predictions enough to move retention measurably in a 2-week A/B, and we'd burn experimentation capacity on a measurement that won't reach significance. The full ladder is the wrong instrument here.'

Calibration ladder

Your team ships a new recsys ranking model. Walk me through the rollout.

Operational reality probe. The interviewer wants to see whether you have the Ladder mental model or whether you'll default to blue/green.

L4 · Mid

Shadow then canary then A/B then ramp. Standard pattern.

Missed: Reciting the standard sequence without thinking about what each step catches. Will run the full ladder on every change including ones that don't need it, and miss the calibration drift the wrong rollout pattern doesn't catch.

L5 · Senior

Same sequence with timing: shadow for 2-3 days, canary at 5% for 24-48 hours, A/B at 50/50 for 2 weeks, then ramp. Each step has a specific check.

Missed: Knew the sequence with durations. Missing the meta-move that the sequence isn't fixed; the steps are dimension-coverage tools.

L6 · Staff

Depends on the change. A new ranker touches model behavior and likely performance, so I'd run shadow, canary, A/B, and keep a permanent holdback. If this were a config-only change touching just one dimension of the regression surface — say a re-rank policy update — I'd skip shadow and go straight to canary. The default 'always run the full ladder' burns engineering time on dimensions the change doesn't touch.

Missed: Strong dimension-based reasoning. Missing the connection to other surface-decomposition patterns from the course (Latency Anatomy, hidden-fork from CLARO) that would land this as a transferable pattern.

L7 · Principal

Before the rollout I want to name which dimensions of the regression surface this change actually touches. A new ranker touches model (2), almost certainly performance (4), probably business (5), possibly integration (3) if it changes output format, rarely data (1). So the mandatory steps are shadow with per-class output distribution monitoring (covers 2 and 4), canary with quality dashboards (confirms 2 and adds 3), A/B with business metric and permanent holdback (covers 5). The order matters because each step's blind spot is the next step's job — A/B alone can't catch a calibration drift that affects all variants equally; only shadow's distribution comparison can. The pattern: rollout steps are coverage tools, and the diagnostic question is 'what does each one actually cover that the others don't?' Same shape as the Latency Anatomy decomposition from Lesson 2.1 — refuse the single-instrument framing, name the surface, pick the instruments that cover it.

What scored L7

Named the rollout as coverage over a regression surface (not a sequence), specified which dimensions a recsys change touches and why, identified the per-step blind spot that the next step covers, and connected it to the surface-decomposition pattern from earlier lessons. The L7 move is treating the rollout ladder as a multi-instrument diagnostic, not a default procedure.

Pattern recognition

When you see

Someone proposes deploying a new model directly to ≥25% of traffic.

→

Think

Stop. The Ladder's lower steps catch a class of failure that ≥25% deployment cannot. Going to 25% without shadow and canary trades 21% of users against the cost of 3 days of slow rollout.

The 'we're confident, let's ship to half' move is the canonical pre-incident move. The model team is confident because offline metrics look great; the offline metrics overstate online performance for the reasons in Lesson 3.1 (point-in-time leaks, feature skew, distribution shift); the rollout exposes the gap to half of users; the regression is invisible for a week because there's no per-version quality observability. Every step the Ladder skips is risk you carry on real traffic. The slow rollout is the cheap version of insurance against the failure mode you don't know exists yet.

Real-world reference · Netflix

Experimentation Platform (XP) and Quasi-Experimentation

Netflix's Tech Blog has published multiple posts on their experimentation infrastructure, including how they decide which experimentation method to use for which class of change. They explicitly distinguish between rollout patterns based on what dimension of regression is being tested — model changes get one pattern, recommendation changes another, infrastructure changes a third. This dimension-aware approach matches the Regression Surface framework from this lesson: the rollout pattern follows from what the change touches.

Takeaway: The Netflix posts demonstrate something most rollout guides skip: the rollout pattern is itself a design choice, not a fixed sequence. Netflix explicitly chooses different patterns for different change classes (model vs UI vs infrastructure) because each touches a different subset of the regression surface. When defending dimension-aware rollout decisions in interviews, citing Netflix's approach demonstrates that the 'always run the full ladder' default is the wasteful version; the right version is matching the rollout pattern to the regression dimensions the change actually touches.

Netflix Tech Blog — Experimentation Platform ↗

Drill · 12 minutes

Practice this. Time yourself.

You have 12 minutes. A team wants to roll out a new ranker that 'looked great in offline eval — 5% better on the primary metric.' They propose going directly to 100%. Walk them through the Ladder, naming what each step would catch that they're proposing to skip. Then propose the timeline they should commit to. Write 4 paragraphs: (1) the failure modes a 100%-direct rollout would expose them to, (2) the catch-cost per Ladder step, (3) the timeline, (4) the observability prerequisite.

Self-assessment rubric

Dimension	Weak	Passing	Strong	Staff bar
Direct-rollout failure modes	Said 'they might have a regression.'	Named 2-3 specific failure modes.	Named: model loading errors, latency regression, output distribution shift, offline-online metric divergence (the recurring failure mode from Lesson 3.1).	Same plus: named that 'looked great in offline' is the canonical pre-incident signal, because offline metrics overstate online performance routinely for structural reasons.
Catch-cost per step	Said each step 'catches problems.'	Per-step catch with one example.	Per-step catch with specific failure class and 'what would have happened on direct rollout' counterfactual.	Same plus: explicit cost-benefit per step. Shadow costs 3 days and catches gross failures at zero risk. Canary costs 2 days and catches quality regressions on 2% of traffic instead of 100%. Each step is justified as insurance with a price tag.
Timeline commitment	Vague 'a few weeks.'	Total timeline of ~3 weeks.	Per-step duration: 3 days shadow, 2 days canary, 14 days A/B, then ramp.	Per-step duration AND the criteria for advancing to the next step (e.g., 'advance from canary if quality metric within 1% of current production and no per-class regression detected').
Observability prerequisite	Did not name.	Said 'we need monitoring.'	Named per-version quality metrics as the prerequisite for the Ladder to work.	Per-version observability AND distribution-shift detection AND quality-regression alerting at the per-class level. Without these, the Ladder is theater.

Reveal model solution

Direct-rollout failure modes. The 'looked great offline' framing is the most common precursor to an online regression. Specific failure modes the team is volunteering for: (1) Model loading errors — the new model artifact has a different schema or fails on the production hardware; 100% deployment means 100% of users see errors. (2) Latency regression — the new model has higher inference cost than measured offline; production p99 violates SLA. (3) Output distribution shift — the model's outputs are systematically different from the old model in ways that break downstream consumers (cache invalidation, re-rank assumptions). (4) Offline-online metric divergence — the 5% offline improvement is partially or entirely a measurement artifact (point-in-time leak, feature skew, distribution shift between training and serving cohorts). Each of these is a known failure mode and each is roughly an order of magnitude cheaper to catch in shadow or canary than in production. Catch-cost per step. Shadow (3 days, zero user impact): catches the first three failure modes — loading errors, latency, distribution shift — at zero risk. Cost: 3 days of dual-serving compute. Counterfactual: any of these in direct rollout is a same-day Sev-1 incident. Canary (2 days, 2% traffic): catches quality regressions on real users at 50× lower exposure than direct rollout. Cost: 2 days of running two models on production traffic. Counterfactual: a 4% regression on 2% of users costs the same as a 0.08% regression on 100% — the canary buys two orders of magnitude in trade-off. A/B (14 days, 50/50): catches the offline-online metric divergence by measuring online effect directly. Cost: 14 days of split traffic with rigorous measurement. Counterfactual: shipping to 100% with no A/B means the team cannot distinguish 'new model is better' from 'new model is worse but seasonal effects are masking it.' Timeline. Day 0 to 3: shadow. Advance if: model loads correctly, p99 latency within 10% of current production, output distribution shift below 0.1 KL on labeled set. Day 3 to 5: canary at 2%. Advance if: quality metric within 1% of current production AND no per-input-class regression beyond noise threshold. Day 5 to 19: A/B at 50/50. Advance to ramped rollout if: primary metric improvement is statistically significant AND guardrail metrics within pre-negotiated trade ratio. Day 19 to 25: ramp from 50% to 100% in stages with a permanent 2% holdback retained on the old model. Total ~25 days; team can compress if specific risks are explicitly accepted, but the compression should be a documented decision with named accountability. Observability prerequisite. None of this works without per-version quality metrics. Specifically: every prediction tagged with model_version, quality proxies (LLM-as-judge sample for generation tasks, downstream engagement for recsys) reported per model_version, distribution-shift alerts on input feature distributions per model_version, per-input-class quality breakdowns to catch slice-specific regressions that aggregate metrics hide. If the team's observability does not include these, the Ladder is theater — they will advance through steps based on the absence of obvious failures, not on the presence of quality signal. The right answer to a team without per-version observability is 'we build that first, then we deploy.' The week of observability work pays back on the first non-trivial rollout.

Common failures

✗Compressed the timeline to 'a few days' to please the team's urgency. The Ladder's duration is the cost of insurance; compressing it is volunteering for risk you don't have to take.
✗Did not connect to per-version observability. Without it, the Ladder steps are theater.
✗Treated the 5% offline lift as evidence of safety. Offline metrics overstate online performance routinely.
✗Did not propose the permanent holdback. Without it, retention effects are unmeasurable.

Artifact · checklist

The Quality-Aware Rollout Checklist

Step 1 — Shadow (Days 0-3)

☐New model serving real production traffic in parallel; outputs logged but not served.
☐Monitor: model loads correctly, p99 latency vs current production, output distribution KL divergence on labeled set.
☐Advance criterion: latency within 10%, KL divergence < threshold, no errors.

Step 2 — Canary (Days 3-5)

☐New model serves 2% of traffic, ideally to low-stakes population.
☐Monitor: per-version quality metrics, per-input-class regression, error rate.
☐Advance criterion: quality within 1% of current AND no per-class regression beyond noise threshold.

Step 3 — A/B (Days 5-19)

☐50/50 split. Primary metric and guardrails measured.
☐Willingness-to-trade ratio between primary and guardrails pre-negotiated with product before A/B starts.
☐Advance criterion: primary metric improvement statistically significant AND guardrails within ratio.

Step 4 — Ramped + holdback (Days 19+)

☐Ramp to 100% in stages (10% → 25% → 50% → 100% over a week).
☐Permanent holdback (1-5% of users) retained on old model for long-term comparison.
☐Holdback is an ongoing operational commitment; assign owner.

Prerequisite — per-version observability

☐Every prediction tagged with model_version.
☐Per-version quality dashboards at same fidelity as latency.
☐Distribution-shift alerts on input features per version.
☐Per-input-class quality breakdowns visible without ad-hoc queries.

Post-mortem · anonymized

Setup

Large e-commerce platform. Recommendation ranker team had a clean rollout discipline — shadow, canary, A/B, ramp — for two years. A new tech lead joined and proposed compressing rollouts to ship faster. The proposed compression: skip shadow when 'offline metrics look great' (defined as >3% lift on primary metric).

What happened

Six weeks after the compression policy went into effect, a new ranker shipped through skip-shadow → canary at 5% → straight to 50% A/B. Canary monitoring caught nothing visible. The A/B showed the new ranker was 2% better on aggregate. Three weeks into the A/B, an enterprise customer reported that their product detail pages were sometimes showing related products from a completely different category. Investigation revealed the new ranker had a subtle output schema change that downstream caching had been silently dropping for 8% of slates. Shadow would have caught this in 24 hours by comparing output distributions. The compression policy cost roughly two weeks of investigation, a customer escalation, and a full rollback.

The moment

The post-incident review identified that the skip-shadow policy had been justified as 'we're confident in our offline metrics.' The metrics didn't lie; the offline pipeline didn't measure schema compatibility because it had no reason to. The failure mode was exactly the kind shadow exists to catch — invisible to quality metrics, visible only to output-distribution monitoring. The tech lead's 'ship faster' policy had removed the step that catches the cheapest failures cheapest.

What they should have said

When the tech lead proposed the compression: 'Each Ladder step exists to catch a specific class of failure that the next step cannot catch as cheaply. Shadow catches schema, loading, latency, distribution-shift failures at zero user risk and 3 days of compute. Canary catches quality regressions at 2% user risk. Skipping shadow saves 3 days and trades zero-user-risk insurance for 2%-user-risk insurance on the same class of failure. The math doesn't work — we're paying with users instead of compute. The compression I'd support is in A/B duration when the primary metric moves fast (recsys engagement signals stabilize in 7-10 days); shadow is the wrong step to compress.'

Lesson

Each step of the Quality-Aware Rollout Ladder catches a different class of failure at a different cost. Compressing the wrong step trades a cheap failure category for an expensive one. The Staff move in 'ship faster' conversations is to name which step compresses safely (usually A/B duration when metrics stabilize quickly) and which step is non-negotiable (usually shadow, because the failures it catches are invisible elsewhere). Treating the Ladder as a fixed pipeline misses where the real flexibility lives.