Real-Time Recommendation Engine (200M users, 100 ms p99) — Simulated Interview

Design a real-time recommendation system for a video platform. 200M MAU, 50M DAU, 100M items, ~1B watch events per day. Three rec slots — homepage, end-of-video, search. p99 100ms SLA on each. You have 45 minutes. Ask whatever you need to.

Before I sketch anything I want to lock down five things — constraints, latency budget, access patterns, read/write shape, and the objective. About four to five minutes. I want to make sure the architecture I draw is the one that fits, not the one I'm familiar with.

C, constraints. Three questions. One — are the three rec slots served by one ranker and one feature pipeline, or are they three separate systems? Two — privacy/compliance regime — GDPR, CCPA, kids' content, anything that restricts cross-slot data sharing? Three — what's the existing infra I should plan around, or is this greenfield?

One ranker, three slates — same model, different candidate sources and slate-specific re-rank rules. CCPA applies; otherwise unrestricted. Greenfield within the platform infra — feature store and online inference platform exist.

Got it — so one ranker shared across slots, with slot-specific candidate generation and re-rank rules. That means the model architecture and training pipeline are shared, and the design splits at the candidate sources and at re-rank. CCPA noted — I'll plan for user-level opt-out paths in the feature store and don't fan user data across regions inappropriately.

L, latency budget. 100 ms p99 on each slot. Let me sketch the budget tree out loud — network plus auth at the edge is 10 to 15ms typically, leaves us 85 to 90ms for the rec decision. That has to fit candidate retrieval, coarse rank, feature lookup, fine rank, re-rank, and response framing. I'd allocate roughly 15 for retrieval, 10 for coarse rank, 25 for feature lookup, 20 for fine rank, 10 for re-rank, 5 for response. We can revisit the split. The thing I want to flag now: this budget forces a two-stage ranker. A single ranker over a thousand candidates does not fit. So the architecture is partially written.

A, access pattern. Three questions. One — what's QPS per slot at peak versus steady? Homepage rec is a load page; end-of-video is fired every video completion; search rec runs on every search query. Those have very different shapes. Two — what fraction of traffic is logged-out or first-session? Drives the cold-start budget. Three — is there a near-real-time signal loop — like, when a user watches a video, do we want subsequent recommendations within the same session to know about it?

Homepage peak 80k QPS, steady 30k. End-of-video peak 200k QPS — every video end. Search rec peak 40k. Cold or first-session traffic is about 12% of slots. Yes, we want near-real-time — within-session signal must reach the next rec within 30 seconds.

200k peak on end-of-video is the dominant number — that's where the GPU sizing and feature store QPS get sized to. 30-second within-session signal means a streaming pipeline, not nightly batch — I'll commit to that now. 12% cold-start is high enough that I cannot route it through the same retrieval path and pretend; we need a separate cold-start candidate source. Three concrete commitments out of one answer.

R, read/write asymmetry. Reads at peak across all three slots are ~320k QPS. Watch events for the write path are 1B/day, ~12k/sec average, probably 50k/sec at peak — much smaller than reads but heavy because each event triggers downstream feature updates. The asymmetry is around 6 to 1 reads versus writes, but the writes are the side that feeds the model. So the system is read-dominated for the serving path and write-dominated for the feature freshness path. I'll treat those as two separate pipelines with different SLAs.

O, objective. This is the one I want you to give me. The interviewer prompt didn't include it. Are we optimizing for watch time, completion rate, daily-active retention, subscription survival, ad revenue? Each of these maps to a different multi-task ranker head ensemble and a different re-rank policy. The 'right' system is fundamentally different for each.

Primary metric is 28-day retention. Engagement metrics are guardrails — pCTR, watch time, no-action rate. Re-rank has hard policy filters: misinformation, kids' content, region restrictions. The team is willing to accept up to 0.5% retention regression for a 5% guardrail improvement, no more.

OK so primary is 28-day retention, guardrails are engagement and policy, and the team will accept 0.5% retention regression for 5% guardrail gain. That trade ratio is the policy I'll design re-rank against. State of the system after CLARO: one shared ranker with slot-specific candidate sources and re-rank; latency-budget-forced four-layer spine; streaming feature pipeline for within-session signal; explicit cold-start path for the 12% of traffic that has no history; primary objective is 28-day retention with named guardrails and a documented trade ratio. I have not drawn a box yet. Let me draw the spine now and walk through each layer.

Tell me the most important failure mode of this design.

Silent drift in the training-data pipeline. If position-bias correction misses a subtle shift — say, the previous policy started favoring a new content category and we don't account for it — the model trains on biased impressions and the bias gets amplified at each iteration. There's no alert that fires; the offline eval looks fine because it's on the same biased distribution. Detection happens when 28-day retention drops in the long-term holdback group three to four weeks later. By then we've shipped four model iterations. The system architecture makes this more likely because we're retraining daily and shipping fast. The mitigation is the input-distribution-drift release gate from Phase 3 — but that's the architecture's most fragile dependency.

Walk me through what happens if the ANN index becomes corrupted — say a bad nightly rebuild ships.

Three layers of defense. (1) Pre-deploy validation: every nightly rebuild gets shadow-evaluated on a labeled retrieval set before the swap. Recall@1000 against the previous index has to be within a tolerance band. If it fails, no swap. (2) Atomic swap with retention: we keep the previous index hot for 24 hours so we can roll back in under a minute. (3) Detection: if recall-quality metrics drop after swap — measured by downstream fine-rank score distributions, not direct labels — we auto-rollback. The bad rebuild ships to disk but doesn't ship to traffic; that's the architectural commitment. The reason this matters is that ANN-index regression is invisible in normal latency metrics — the system is fast and wrong, not slow and right.

How do you decide when to add another candidate retrieval source?

It's a budget question, not a capability question. A new retrieval source costs roughly a millisecond of the 15ms retrieval budget and a recurring cost in feature engineering, training, evaluation, on-call. The decision criterion is: can this source earn its budget — that is, can it contribute a measurable fraction of impressions to the final fine-ranked slate, on a labeled set, beyond what existing sources cover? If yes, ship and instrument. If we can't prove it beats the existing portfolio, we don't add it. The corollary: most teams don't decommission sources that are no longer earning their budget. That's the bigger problem in mature recsys — sources accumulate. I'd commit to a quarterly review where any source not earning its budget gets decommissioned.

What's the on-call runbook for a quality regression — say, a Sev-2 'feed is showing repetitive content'?

Triage in three steps. (1) Which layer? — check the per-layer dashboards from Phase 3. Re-rank diversity dashboard is the first place to look for repetition; if MMR or rolling diversity has shifted, we know the cause is policy. If diversity looks normal, drop to fine rank — has a category score head shifted? If fine rank is fine, drop to retrieval portfolio — has one source's contribution spiked? (2) What changed? — correlate against the last 24 hours of pushes: model rollouts, feature-store schema changes, policy updates. (3) Rollback or mitigate? — if a recent change caused it, rollback first and investigate after. If no recent change, deploy a temporary policy mitigation in re-rank (forced diversity boost) while we debug the root cause. The runbook is short on purpose — three steps, observable from the same dashboard, with a default of 'rollback first.' Most quality regressions in recsys are caused by recent pushes, not by external shifts.

If I gave you 10% fewer GPUs, what would you cut first?

I'd take it out of fine-rank candidate count first. Going from 200 to 175 candidates is roughly a linear GPU saving in fine rank and feature lookup — about 12% capacity recovered for a small recall@200 hit that the re-rank layer can mostly absorb because the top 50 of the fine-ranked slate is what actually matters for the slate. That's the cheap cut. I would not cut at the retrieval layer first because retrieval recall is what determines what the ranker can do; cutting there caps the system's ceiling. I'd also explicitly not cut the degraded-mode model capacity, because that's the path that runs during failure and the failure costs 10× the steady-state cost. The pattern: when cutting capacity, cut where the marginal quality loss is smallest and the failure modes are smallest, not where the headline GPU cost is biggest.