Multi-Modal Content Moderation (Meta-scale) — Simulated Interview
Meta-scale content moderation is two systems with two SLAs and a human-review feedback loop, not one pipeline that does everything. The Cascaded Sync + Streaming Async framework names the decomposition and forces the candidate to address the structural fact that policy categories have different latency budgets.
Content moderation interviews are the largest, messiest design problem in the Staff loop. The scale is enormous (1B+ posts/day for top platforms), the categories number in the dozens (and have policy-specific definitions), the modalities span text, image, video, and live streaming, and the policy team has hard requirements about what cannot go live and softer requirements about what must come down within a window. Most candidates respond to this complexity by designing one pipeline that handles everything badly. The Staff move is to recognize that the policy structure itself decomposes the system: some categories cannot be allowed online for any time at all, others can be live for 60 seconds, others for 5 minutes. The latency tiers force two coupled systems, not one.
The Cascaded Sync + Streaming Async pattern is the structural answer. The sync path is a fast cascaded classifier that gates pre-publish for unconditional-block categories; the async path is a streaming detector that scans everything else within a policy-committed window; the human review queue is the third system that catches borderline cases and feeds labels back to training. Each path has its own quality bar, its own cost curve (text cheap, image 10-30× cost, video 50-200× cost), and its own failure mode. Designing only the sync path means harmful content sits in users' feeds for minutes; designing only the async path means CSAM goes live for the duration of the async window. The two systems together are the design.
Cascaded Sync + Streaming Async
Content moderation at Meta-scale is two systems with two different SLAs, not one system pretending to do both. The Cascaded Sync + Streaming Async pattern names the structural decomposition: a fast cascaded sync path that gates pre-publish for the small set of unconditionally-blocked categories, and a streaming async path that scans everything else within a policy-committed time window. Most candidates design one pipeline. The right answer is two coupled systems with sharply different latency budgets and quality bars.
- 1Sync path — Cascaded classifier (200 ms p99)Pre-publish gate for the categories that cannot be allowed online for any time at all — CSAM, terror content, copyright known matches, immediate-imminent violence. Small fast model handles 85-90% of clear cases; large second-pass model handles uncertain residual. The architecture is a cascade because a single big model violates the SLA at the QPS volumes a publishing service generates.
- 2Async path — Streaming detection (60 s p99 for high-severity, 5 min for medium)Post-publish scanner for everything that didn't trigger sync block. Runs on Kafka stream of newly-published content with a richer model and the full feature context. The 60-second budget for high-severity is short enough to require streaming infrastructure, not batch — anything slower would mean harmful content is live for minutes, which is a policy commitment most platforms cannot make.
- 3Multi-modal: text + image + video, with different cost curvesText moderation is cheap (~$0.0001/post inline); image is 10-30× more expensive; video is 50-200× more expensive depending on length. The cost model is dominated by the media-bearing slice of traffic, which is usually 20-40% of posts. Designing assuming uniform cost per post produces budget overruns of 10x.
- 4Human review queue — the third systemBorderline cases from both sync and async paths route to human reviewers with a separate SLA (typically hours to days). The review queue is itself a system with capacity-planning, reviewer-quality monitoring, and feedback to model training. Treating human review as 'just append to a queue' misses that queue depth, reviewer well-being, and label drift are all system properties to monitor.
- 5Feedback loop — reviewer labels become training dataThe reviewer labels are the highest-quality training data the moderation team has access to. The right architecture is for the labels to flow back into the training pipeline as a continuous feedback loop, with deliberate sampling to avoid reviewer-bias amplification. Most production moderation systems either underuse this signal (treating review as a dead-end) or overuse it (overfitting to reviewer disagreement patterns).
Apply the framework to any content moderation, abuse detection, or pre-publish policy enforcement interview. Also applies to fraud-of-content detection (fake reviews, fake accounts, fake media) where the same sync/async split applies with different category cuts.
Senior answer to 'design Meta-scale moderation': 'Train a multi-task classifier on posts, deploy at scale, monitor for drift.' Staff answer: 'Two systems. Sync gate at 200 ms p99 with cascaded classifier — small model for 85% of clear cases, big model second-pass for uncertain residual, blocking pre-publish only for unconditional categories. Async scanner at 60 s p99 for high-severity, 5 min for medium, on Kafka stream with a richer model. Plus human review queue with feedback loop into training. Multi-modal cost model: text cheap, image 20× cost, video 100× cost; media-bearing posts dominate the budget. Each layer has different precision/recall targets and the willingness-to-trade ratio is negotiated separately with the policy team for each.'
What's your strategy for handling the long tail of policy categories — say, 80 categories total, most low-volume but each with specific definitions and trade-offs?
Most candidates default to 'multi-task model' without thinking about how 80 categories actually compose into a serving system.
Train a single multi-task model with 80 heads, deploy as one system.
Group categories by similarity and train cluster-specific models. Combine outputs with a calibration layer. Add per-category thresholds tuned to that category's willingness-to-trade ratio.
Three-tier composition. Tier 1: 5-10 high-volume categories (spam, NSFW, common harassment) — share a multi-task model trained on large data; serves all traffic. Tier 2: 20-30 medium-volume categories — separate models per category cluster, trained on lower-volume data with explicit handling for class imbalance. Tier 3: long tail of rare-but-important categories (specific extremism, region-specific policy) — rule-based or small classifiers that operate as filters with explicit policy-team ownership per category. Each tier has different training cadence, deployment process, and accountability.
Same three-tier composition with two meta-additions. (1) The tiering itself is a policy-engineering co-design, not an engineering decision alone. Policy teams own which categories are unconditional-block (Tier 0, sync path), which are async-detectable (Tiers 1-2), and which are 'monitor but route to human first' (Tier 3). The engineering team owns the latency and cost trade-offs within each tier. The boundary between policy decisions and engineering decisions is explicit and re-negotiated quarterly. (2) The most important architectural property: the long-tail categories can evolve independently of the high-volume ones. Adding a new long-tail category (e.g., a new regional disinformation pattern) should not require retraining the multi-task model — it should be a plugin to Tier 3 with its own deploy cycle. This is what lets the moderation system adapt to new threats on the timescale they emerge (days), not the timescale full retraining allows (weeks). Most production moderation systems are too monolithic precisely because the team treated 80 categories as one ML problem instead of 80 separately-evolvable policy decisions. The pattern: when scope is heterogeneous, the architecture must support independent evolution of the heterogeneous parts. Same shape as the retrieval-portfolio pattern from the recsys lesson.
Named that long-tail categories must evolve independently of high-volume ones, and that the policy-engineering boundary is itself a design artifact that needs explicit negotiation. The pattern of 'heterogeneous scope requires independently-evolvable architecture' is portable to any multi-domain system.
Meta-scale moderation. Two coupled paths (sync gate + async scanner) plus human review and training feedback. Notice the policy categories are tiered, the modalities are routed differently due to cost, and the reviewer labels flow back into training as a continuous loop.
Practice this. Time yourself.
You have 12 minutes. Your moderation system handles 50k posts/sec with text only at p99 200 ms sync, 60 s async. Product wants to launch video. Video posts will be 5% of volume. Each video moderation takes 8 seconds and costs $0.015. Walk through the design changes. Write a 4-paragraph response: (1) cost impact analysis, (2) why video cannot serve the same SLA as text, (3) the architectural decoupling required, (4) what you'd push back on with the product team.
Self-assessment rubric
| Dimension | Weak | Passing | Strong | Staff bar |
|---|---|---|---|---|
| Cost analysis | 'Will cost more.' | 5% of 50k QPS × $0.015 = $37.5/sec = $3.2M/day for video alone. | Same plus: per-post cost comparison shows video at 150× text cost; video dominates the total budget. | Same plus: framed as 'video at 5% of volume will be ~95% of moderation spend; need to negotiate budget with product before this lands.' |
| Why video can't serve text SLA | 'Video is slower.' | 8 s per video doesn't fit 200 ms sync SLA. Period. | Same plus: video cannot serve pre-publish gate for any category; must use a different policy commitment for video — typically 'publish, then async-detect within minutes.' | Same plus: even Tier 0 unconditional-block categories for video must accept some live exposure window because pre-publish is infeasible; this is a policy negotiation, not a technical one. |
| Architectural decoupling | 'Use a separate model.' | Separate video moderation service with its own SLA tier. | Same plus: routing layer that dispatches by modality, separate Kafka topic for video stream, separate model service with GPU sizing for video workload. | Same plus: explicit modality-tiered policy — video gets 'detected within 60 s post-publish' commitment, not 'gated pre-publish.' Engineering and policy aligned on the new SLA before launch. |
| Pushback on product | No pushback. | Asked for budget approval. | Specifically: 'video adds $3.2M/day in moderation cost; product needs to confirm budget AND agree to weaker pre-publish gate for video.' | Same plus: proposed cost-reduction levers (sample only borderline videos for full scoring; thumbnail-only first-pass) and asked product to commit to which categories truly require video-level moderation vs which can use cheaper signals. |
Reveal model solution
Common failures
- ✗Did not compute the cost impact in dollars. Cost is the binding constraint here.
- ✗Treated video as a 'slower text' problem. The architectural decoupling is non-negotiable.
- ✗Did not flag the policy implication (video cannot pre-publish gate). Engineering can't make this commitment alone.
- ✗Did not propose cost-reduction levers. Pure 'add capacity' answers are the canonical Senior failure.
The Moderation Tier Design Worksheet
Step 1 — Categorize the policy tiers
- ☐Tier 0 — unconditional block (CSAM, terror, etc.): sync pre-publish gate; SLA = 200 ms.
- ☐Tier 1 — high-severity post-publish: streaming async; SLA = 60 s.
- ☐Tier 2 — medium-severity post-publish: streaming async; SLA = 5 min.
- ☐Tier 3 — long-tail evolving categories: plugin filters; deploy independently of main models.
Step 2 — Modality routing
- ☐Text: cheap, fits all tiers including sync.
- ☐Image: 10-30× cost; fits async, sampling for Tier 2.
- ☐Video: 50-200× cost; cannot serve sync; explicit policy on live-exposure window.
- ☐Live stream: separate system entirely; sampling and reactive takedown only.
Step 3 — Cost model
- ☐Per-post cost per modality.
- ☐Volume × cost per modality = daily spend.
- ☐Identify the dominant modality (usually media) — it determines budget.
- ☐Negotiate budget with product before launch, not after.
Step 4 — Human review and training feedback
- ☐Reviewer capacity sized to borderline volume per tier.
- ☐Reviewer SLA: hours for high-severity, days for medium.
- ☐Reviewer labels flow back to training as a continuous loop with bias-aware sampling.
- ☐Reviewer well-being and label-drift are monitored as system properties.
Step 5 — Policy-engineering co-design
- ☐Tier definitions owned by policy team.
- ☐Latency budgets owned by engineering.
- ☐Boundary re-negotiated quarterly as threats evolve.
- ☐Long-tail categories deploy independently of high-volume models.
Mid-large social platform launching video posting. Engineering team built moderation for text + image and treated video as 'phase 2.' Product launched video at 5% expected volume without confirming with the moderation team.
Video posts hit production and the moderation team had no path to scan them. For three weeks, all video moderation was reactive — content was up until a user reported it. A creator-facing news story broke about objectionable video content remaining live for days. Trust & Safety leadership escalated to the CTO. The engineering team scrambled to build a video moderation pipeline in two weeks, with significant cost overruns ($4M unbudgeted) and an architecture that was a thin wrapper around existing image moderation rather than a properly designed video pipeline. The wrapper remained in production for nine months before being properly replaced.
The retrospective identified that the launch decision had been made without consulting moderation. Product had treated 'we have moderation' as a binary; engineering had treated 'we'll add video later' as deferrable. Neither team had named that video moderation is structurally different — different SLA, different cost curve, different policy commitments — and the design conversation needed to happen before launch, not after.
Six months before launch, in the initial product spec review: 'Video moderation is a structurally different system from text/image. Cost is ~150× per post; SLA cannot be pre-publish; policy commitments will weaken. We need: budget approval for ~$3-5M/day at expected volume, policy team agreement on the new SLA, six months to build the dedicated video moderation pipeline, and a launch plan that ramps video volume in step with moderation capacity. Without all four, we should not launch video.' That conversation, made early, would have produced either a delayed launch or a properly designed system at launch. Neither happened because no one in the room was framing video moderation as a separate system.
Multi-modal moderation is not 'one pipeline that handles more modalities.' It is separate systems with separate SLAs, separate costs, and separate policy commitments. The Cascaded Sync + Streaming Async framework forces this decomposition by naming the latency tiers and the cost curve per modality. The Staff move at design time is to refuse the 'add modality later' framing and to demand the policy-engineering co-design conversation before the product team has committed to a launch date.