AI Mock Interview Platform: System Design at Scale

The prompt looks friendly: "Design an AI platform that runs mock interviews for a million candidates." Then you start drawing boxes and realize this is not a CRUD app with a chatbot bolted on.

An interview is a long, stateful, real-time conversation. The system has to listen, think, and speak back fast enough to feel human, hold that session open for 30 to 45 minutes without dropping it, run untrusted candidate code safely, and then grade the whole thing in a way you can defend to a hiring manager or a regulator. The expensive, interesting parts are latency, statefulness, GPU cost, and fair scoring, none of which show up in a typical "design a URL shortener" answer.

This is the answer I would give at a staff-level system design interview, and the design I would actually build.

Quick Answer

Model each interview as a stateful, long-lived session driving a low-latency voice loop: streaming speech-to-text, a fast interviewer LLM, and streaming text-to-speech under an 800 ms turn budget. Serve models on GPUs with continuous batching, checkpoint transcripts for failover, score asynchronously with a rubric-based LLM judge, and size for peak concurrent sessions, not signups.

TL;DR

• Size from peak concurrent sessions. One million monthly users is roughly ten thousand concurrent live interviews at peak, not one million.
• The core loop is voice-to-voice: capture → end-of-turn detection → streaming ASR → interviewer LLM → streaming TTS → playback, targeting under ~1 second.
• Treat each session as a stateful actor on a worker, with the transcript checkpointed to durable storage so a crash never loses the interview.
• Sessions are mostly idle. GPU demand follows active generations, so continuous batching plus warm GPU pools keep cost sane.
• Separate models by job: a fast interviewer model for the live loop, a slower, smarter judge model for async scoring.
• Score from an explicit rubric with cited evidence, temperature 0, and multi-sample aggregation, calibrated to a human-labeled gold set.
• Run candidate code in per-execution microVMs with no network and hard limits.
• Voice is biometric data and automated scoring can trigger hiring-bias law. Consent, retention, and bias audits are part of the design.
• Fail gracefully: degrade voice to text, fail over models, and never drop a session silently.

What Most "Design a Mock Interview" Answers Miss

Search results and generic answers tend to draw a client, an "AI service," and a database, then move on. That skips every part an interviewer actually probes.

The gaps are almost always:

• The latency budget. Where the ~1 second voice-to-voice response actually goes, and why end-of-turn detection, not the model, is usually the bottleneck.
• Statefulness. Interviews are long-lived sessions, so sticky routing, reconnection, mid-session failover, and draining during deploys all matter.
• GPU economics. Capacity and cost are driven by active generations, not user count, and GPUs scale slowly, which changes the autoscaling design.
• Scoring reliability. LLM-as-judge is biased and noisy unless you constrain it with rubrics, evidence, low temperature, sampling, and calibration.
• Safe code execution. Running a million candidates' code is a multi-tenant sandboxing problem, not a eval() call.
• Compliance. Voice recordings and automated scoring pull in biometric-privacy and employment-bias regulations.

This answer closes those gaps.

Scope the Interview First

State assumptions before drawing boxes. It shows you know system design is constraint-driven, and it stops you from over-building.

Functional requirements

1. A candidate starts a mock interview by type: coding, behavioral, or system design.
2. An AI interviewer conducts a real-time spoken conversation, asks follow-ups, and adapts difficulty.
3. Coding interviews include a shared editor and safe code execution against tests.
4. The session produces a transcript, a rubric-based score, and written feedback.
5. Candidates can review past sessions, scores, and progress over time.
6. Content is drawn from a versioned question bank with anti-repetition.

Non-functional requirements

• Scale: 1,000,000 monthly active candidates; design for 10,000 concurrent live interviews at peak, with headroom to a 20,000 burst.
• Latency: voice-to-voice under 1.2 s p95; editor and transcript updates under 150 ms.
• Availability: 99.9% monthly for the session control plane; a single model or provider failure must not drop live interviews.
• Durability: never lose a transcript once the interview has started.
• Integrity: scores are consistent, explainable, and auditable for bias.
• Privacy: voice and transcripts are sensitive; consent, retention, and residency are enforced.

These are interview assumptions. Say you would replace them with measured traffic and latency distributions from real usage.

Back-of-the-Envelope: Sizing for One Million Candidates

The number that matters is concurrency, and you have to derive it.

• 1M monthly active users × ~2 sessions each ≈ 2M sessions/month.
• Average session ≈ 30 min → 60M interview-minutes/month = 1M session-hours.
• A month is ~730 hours, so average concurrency ≈ 1M / 730 ≈ 1,370 live sessions.
• Interview practice is bursty (evenings in a few timezones, weekends, pre-hiring season). Apply a peak-to-average factor of ~7×.
• Peak ≈ 10,000 concurrent live interviews. Design for that, burst to 20,000.

Now translate concurrency into resource demand.

The key insight: a session open for 30 minutes is not generating for 30 minutes. The AI speaks in short bursts; the candidate speaks and thinks the rest. So the GPU-bound work is active generations, which is a small fraction of concurrency. That single observation cuts your GPU bill by roughly an order of magnitude versus naive "one model instance per user" thinking.

High-Level Architecture

Separate three planes: a real-time plane that must be fast, an async plane that must be thorough, and a control plane that must be correct. Keep the real-time data path small and deterministic.

Component responsibilities

The Real-Time Interview Loop

This is the part that makes or breaks the product. If the AI takes two seconds to answer, the interview feels robotic and candidates churn. The target is voice-to-voice latency near one second. To hit it, budget every stage.

Latency budget

Two non-obvious points win interview credibility:

1. End-of-turn detection dominates. A naive design waits for a fixed ~700 ms of silence before deciding the candidate is done, which alone blows the budget. Combine a fast VAD with a semantic turn detector that recognizes a completed thought, so you can shorten the silence threshold without interrupting people mid-sentence.
2. Stream everything. Do not wait for the full LLM answer before speaking. Pipe LLM tokens into TTS as they arrive, and start audio on the first sentence. First token, not full response, is what the candidate hears as latency.

Barge-in and turn-taking

Real interviews have interruptions. When the candidate starts talking while the AI is speaking, you must stop the AI immediately (barge-in), or it feels like talking over someone. That means the playback of AI audio has to be cancelable at any moment, and the loop restarts a candidate turn.

// A single interview turn loop. Runs per session on the worker.
// VAD, ASR, LLM, and TTS are streaming interfaces injected for testability.
class InterviewTurnLoop {
  private aiPlayback: AbortController | null = null; // non-null while AI speaks
 
  constructor(
    private vad: VoiceActivityDetector,
    private asr: StreamingAsr,
    private llm: InterviewerModel,
    private tts: StreamingTts,
    private audioOut: AudioSink,
  ) {}
 
  async onAudioFrame(frame: Int16Array): Promise<void> {
    const voiced = this.vad.process(frame);
 
    // Barge-in: candidate speaks while the AI is talking -> cut the AI off.
    if (voiced && this.aiPlayback) {
      this.aiPlayback.abort();
      this.aiPlayback = null;
    }
 
    this.asr.pushAudio(frame);
 
    // Endpointing: only respond once the turn is actually complete.
    if (this.vad.endOfUtterance()) {
      const transcript = await this.asr.finalize();
      if (transcript.trim().length > 0) {
        await this.respond(transcript);
      }
    }
  }
 
  private async respond(candidateText: string): Promise<void> {
    const controller = new AbortController();
    this.aiPlayback = controller;
 
    // Stream LLM tokens straight into TTS; do not wait for the full answer.
    const tokens = this.llm.stream(candidateText, { signal: controller.signal });
    for await (const chunk of tokens) {
      if (controller.signal.aborted) break;
      const audio = await this.tts.synthesize(chunk, controller.signal);
      this.audioOut.enqueue(audio); // first audio should start within ~150 ms
    }
    if (this.aiPlayback === controller) this.aiPlayback = null;
  }
}

Key lines:

• this.aiPlayback.abort() is barge-in. Every downstream stage checks the signal and stops, so the AI goes quiet the instant the candidate speaks.
• this.vad.endOfUtterance() is the endpointing gate. It is the single most important latency and quality knob in the whole system.
• The for await loop pipelines LLM output into TTS chunk by chunk, so audio starts on the first sentence instead of after the last token.

Step-by-step Solution

Step 1: Model each session as a stateful actor

An interview is not a request. It is a conversation with memory. Give every session a single owner: a session worker that holds the live turn state, the running transcript, the current question, and the difficulty pointer in memory, and routes all of that session's traffic to one place.

Use a session registry (Redis) for presence and routing, and checkpoint the transcript to durable append-only storage on every turn. In-memory state is for speed; the durable log is the source of truth. This is what lets you survive a crash without losing the interview.

export type SessionState =
  | "provisioning" | "connecting" | "active"
  | "reconnecting" | "wrapping_up" | "evaluating"
  | "complete" | "failed";
 
export type SessionEvent =
  | { type: "WORKER_READY" }      | { type: "CLIENT_CONNECTED" }
  | { type: "TRANSPORT_LOST" }    | { type: "TRANSPORT_RESTORED" }
  | { type: "INTERVIEW_ENDED" }   | { type: "GRACE_EXPIRED" }
  | { type: "SCORING_DONE" }      | { type: "FATAL"; reason: string };
 
const transitions: Record<SessionState, Partial<Record<SessionEvent["type"], SessionState>>> = {
  provisioning: { WORKER_READY: "connecting", FATAL: "failed" },
  connecting:   { CLIENT_CONNECTED: "active", TRANSPORT_LOST: "reconnecting", FATAL: "failed" },
  active:       { TRANSPORT_LOST: "reconnecting", INTERVIEW_ENDED: "wrapping_up", FATAL: "failed" },
  reconnecting: { TRANSPORT_RESTORED: "active", GRACE_EXPIRED: "evaluating", FATAL: "failed" },
  wrapping_up:  { INTERVIEW_ENDED: "evaluating", FATAL: "failed" },
  evaluating:   { SCORING_DONE: "complete", FATAL: "failed" },
  complete:     {},
  failed:       {},
};
 
export function next(state: SessionState, event: SessionEvent): SessionState {
  return transitions[state][event.type] ?? state; // ignore illegal transitions
}

The design detail interviewers like: GRACE_EXPIRED routes to evaluating, not failed. If a candidate's network dies and never comes back, you still score the partial transcript instead of throwing the interview away. Illegal transitions are ignored rather than crashing the worker.

Expected transition sequence for a clean interview with one dropout:

provisioning --WORKER_READY-->      connecting
connecting   --CLIENT_CONNECTED-->  active
active       --TRANSPORT_LOST-->    reconnecting
reconnecting --TRANSPORT_RESTORED-->active
active       --INTERVIEW_ENDED-->   wrapping_up
wrapping_up  --INTERVIEW_ENDED-->   evaluating
evaluating   --SCORING_DONE-->      complete

Step 2: Choose the voice stack deliberately

There are three ways to build the loop, and the choice is a real tradeoff you should defend, not a detail.

Because scoring is the entire point of a mock interview platform, you almost always need the transcript, which rules out treating a black-box speech-to-speech API as the only path. A common trajectory is to launch hybrid, then self-host the components whose per-minute cost hurts most as volume grows.

Step 3: Serve models with continuous batching

The interviewer LLM is the GPU cost center. Serve it with continuous (in-flight) batching so many sessions' token streams share a GPU efficiently, using a serving stack such as vLLM or NVIDIA Triton. The memory trick that makes this work is paged KV-cache management, which lets the server pack many concurrent, variable-length generations into one GPU.

Practical serving rules:

• Right-size the interviewer model. A mid-size, fast model that gives good follow-ups beats a frontier model that answers in two seconds. Save the big model for async scoring.
• Cache the system prompt. The interviewer instructions and rubric context are identical across turns; prompt caching cuts prefill cost and latency.
• Keep GPUs warm. GPU nodes take minutes to provision and load weights, so reactive autoscaling alone will miss peaks. Use predictive/scheduled scaling plus a warm pool and an admission queue.
• Separate model tiers. ASR, interviewer LLM, TTS, and the judge each scale on their own signal. Do not couple them behind one autoscaler.

Because sessions are idle most of the time, the number of GPUs tracks the ~1,500 to 2,500 active generations, not the 10,000 sessions. If one warm replica sustains ~100 concurrent low-latency decode streams for your interviewer model, you need on the order of 20–30 replicas plus redundancy, not thousands. Treat that as an illustration to confirm with load tests, not a fixed law.

Step 4: Make sessions survive failures

Live model APIs rate-limit and time out. Networks drop. Deploys happen mid-session. None of that can silently kill an interview.

The rule: degrade, do not drop. A text-only fallback interview is worth far more to the candidate than a crash. Wrap every external model call with a timeout, bounded retries with jitter, and a circuit breaker, and keep a second provider configured for the two models on the critical path.

Step 5: Score asynchronously with a rubric and an LLM judge

Do not grade during the live loop; it adds latency and pressure to be fast. When the interview ends, run scoring asynchronously with a stronger model and an explicit rubric. Raw "rate this interview 1–10" prompting is unreliable: LLM judges show position, verbosity, and self-enhancement bias, and low reproducibility. Constrain them.

import * as z from "zod";
 
// Every dimension must cite evidence BEFORE it scores. This grounds the judgment
// in the transcript and makes the score auditable.
const Dimension = z.object({
  evidence: z.string().min(1), // quote or reference specific transcript moments
  score: z.number().int().min(1).max(5),
});
 
export const CodingRubric = z.object({
  correctness: Dimension,   // does the solution actually work?
  complexity: Dimension,    // time/space analysis and choices
  codeQuality: Dimension,   // readability, structure, naming
  communication: Dimension, // did they explain their thinking?
  testing: Dimension,       // edge cases, verification
  overall: z.number().int().min(1).max(5),
  summary: z.string(),
});
export type CodingScore = z.infer<typeof CodingRubric>;
 
const SYSTEM = `You are a calibrated interview evaluator. Score ONLY from the
transcript. For each dimension, first cite specific evidence, then choose the score
whose anchor best matches. Never reward confident but incorrect claims. JSON only.`;
 
export async function scoreCodingInterview(
  transcript: string,
  anchors: string, // explicit descriptions for scores 1..5 per dimension
  judge: JudgeModel,
): Promise<CodingScore> {
  // Sample several independent judgments and take the median per dimension.
  const runs = await Promise.all(
    [0, 1, 2].map(() =>
      judge.complete({
        system: SYSTEM,
        user: `Scoring anchors:\n${anchors}\n\nTranscript:\n${transcript}`,
        temperature: 0,
        responseSchema: CodingRubric,
      }),
    ),
  );
  return aggregateByMedian(runs); // median resists a single outlier judgment
}

Why each control matters:

• Evidence before score forces the model to ground each rating in the transcript, which both improves accuracy and gives the candidate a defensible explanation.
• Explicit anchors (what a 3 versus a 4 means for each dimension) remove most of the vagueness that makes scores drift.
• Temperature 0 plus N-sample median cuts run-to-run variance and outlier judgments.
• A separate judge model avoids the interviewer grading its own conversation.

Expected output shape:

{
  "correctness": { "evidence": "Passed 4/5 tests; missed empty-array case at 14:20", "score": 3 },
  "complexity":  { "evidence": "Stated and achieved O(n log n); justified the sort", "score": 4 },
  "codeQuality": { "evidence": "Clear names; one 30-line function should be split", "score": 3 },
  "communication": { "evidence": "Narrated approach before coding; paused silently 12:00–13:10", "score": 4 },
  "testing":     { "evidence": "Added two edge-case tests unprompted", "score": 4 },
  "overall": 4,
  "summary": "Solid, well-communicated solution; missed one edge case and could decompose one function."
}

Finally, calibrate. Keep a human-labeled gold set of past interviews, measure agreement between the judge and human scorers, and treat a drop in agreement as a regression when you change models or prompts. Version the rubric and the judge model together.

Step 6: Sandbox candidate code execution

Coding interviews run untrusted code from a million people. Every execution gets a throwaway, locked-down sandbox using Firecracker microVMs or gVisor-isolated containers.

{
  "runtime": "firecracker",
  "vcpus": 1,
  "memoryMiB": 256,
  "wallClockTimeoutMs": 8000,
  "cpuTimeoutMs": 5000,
  "network": "none",
  "readonlyRootfs": true,
  "writableTmpfsMiB": 64,
  "pidsLimit": 64,
  "seccompProfile": "runtime-default",
  "droppedCapabilities": ["ALL"],
  "user": "10001:10001"
}

Each field is a control:

• network: "none" stops data exfiltration and SSRF from submitted code.
• readonlyRootfs plus a small writableTmpfs limits what code can persist.
• wallClockTimeoutMs and cpuTimeoutMs kill infinite loops and fork bombs.
• pidsLimit and droppedCapabilities shrink the blast radius of a container escape attempt.
• user: "10001:10001" never runs as root.

Never reuse a sandbox across candidates, and cap concurrent executions so a spike in "run" clicks cannot exhaust the pool.

Step 7: Select questions adaptively

A good interviewer does not read from a fixed script. Store questions in a versioned bank with a calibrated difficulty (an Elo- or IRT-style rating that updates as candidates attempt them). During a session:

• Pick the next question near the candidate's estimated ability, adjusting up or down based on how the last answer went.
• Suppress recently seen questions per candidate to prevent memorization.
• Let the LLM generate follow-ups, but keep the backbone questions human-authored and reviewed so difficulty and fairness stay controlled.

This keeps the interview challenging without being demoralizing, and it produces a signal (estimated ability over time) that candidates value more than a single score.

Step 8: Control cost per interview minute

At 60M interview-minutes a month, the unit that governs the whole design is cost per interview-minute. Even a small number times 60M is a large bill.

Illustrative sensitivity: at 60M minutes/month, an all-in cost of $0.03/min is $1.8M/month; $0.015/min is $0.9M/month. Halving cost-per-minute saves ~$900k a month, which is why model right-sizing and batching are architecture decisions, not tuning. (Numbers are illustrative; confirm with your own rates.)

Common Causes

These are the recurring reasons million-scale AI interview platforms fail, and what each one really is:

1. Sizing by user count. Provisioning for 1M "users" instead of ~10k concurrent sessions wastes money or, worse, under-provisions the burst.
2. A batch mindset in a real-time loop. Waiting for full transcription or the full LLM answer before speaking blows the latency budget.
3. Fixed-silence endpointing. A static ~700 ms silence timer either cuts people off or makes the AI feel slow.
4. Stateless assumptions. Treating a 30-minute session like a request means no reconnection, no checkpoint, and lost interviews on any hiccup.
5. One giant model for everything. Using a frontier model for the live loop makes it slow and expensive; using a tiny model for scoring makes grades unreliable.
6. Ungrounded scoring. "Rate 1–10" prompting produces biased, drifting scores nobody can defend.
7. Shared or reused code sandboxes. Reusing an execution environment leaks state between candidates and widens the attack surface.
8. Reactive-only GPU autoscaling. Relying on HPA for GPUs that take minutes to warm guarantees you miss peaks.
9. Keeping raw audio. Storing every recording invites biometric-privacy liability and a large storage bill for little benefit.

Symptoms

Root Cause

The root cause is almost always a mismatch between the interview's real nature and a request/response mental model.

Capacity and Performance

The platform is a mix of latency-critical, GPU-bound work (the live loop) and throughput-oriented, batchable work (scoring). Design them separately.

• Live loop: optimize for p95 latency, not utilization. Keep headroom; a queued interview turn is a bad experience. Prefer more, smaller warm replicas over fewer saturated ones.
• Scoring: optimize for throughput and cost. Batch it, run it on spare or cheaper capacity, and tolerate minutes of delay.
• Concurrency math: by Little's Law, active generations ≈ arrival rate × average generation time. Keep generation time down (right-sized model, streaming) and the concurrent GPU load drops with it.
• GPU vs CPU: ASR, LLM, and TTS are GPU work; session orchestration, routing, and business logic are ordinary CPU services that scale the easy way. Keep them on separate fleets so a GPU shortage cannot take down session management.

Verification Steps

Latency and quality

• Voice-to-voice p95 under 1.2 s across regions, measured end to end.
• Barge-in stops AI audio within ~100 ms of candidate speech.
• Endpointing does not cut off natural pauses in structured answers.
• First AI audio starts on the first sentence, not the full response.

Reliability

• A killed session worker loses no committed transcript; the client reattaches.
• A dropped client within the grace window resumes; past it, the partial is scored.
• A forced model-provider 429 fails over without dropping the interview.
• A rolling deploy drains or migrates active sessions without mass disconnects.

Scoring integrity

• Identical transcripts produce stable scores across repeated runs.
• Judge-versus-human agreement on the gold set stays above threshold.
• Score distributions show no unjustified gap across demographic groups.
• Every score carries transcript evidence a human can review.

Security and cost

• Submitted code cannot open a network connection or persist outside tmpfs.
• Infinite loops and fork bombs are killed within the configured limits.
• Cost-per-interview-minute is tracked and alerts on regression.
• Raw audio retention matches the consented, documented policy.

Troubleshooting Matrix

Platform and Deployment Notes

Cloud and GPU

Run GPU model servers in an autoscaling pool with a warm buffer sized to your burst, because provisioning and weight-loading take minutes. Use scheduled scale-ups ahead of known peaks (evenings, recruiting season) and an admission queue so a burst degrades gracefully instead of erroring. Keep model weights on fast local or cached storage to cut cold-start time.

Docker and Kubernetes

Put ASR, LLM, TTS, judge, and session workers in separate deployments with their own HPA or custom autoscaler and their own resource classes. Session workers need graceful draining (finish or migrate active interviews on shutdown), so set generous termination grace periods and handle SIGTERM by pausing and checkpointing.

Media edge

Terminate WebRTC at regional edges near candidates to keep RTT low, and bridge audio to the model plane over your private network. Use an SFU pattern (forward streams) rather than mixing, and fall back to a WebSocket transport for text-only and coding interviews.

CI/CD

Gate deploys on evaluation suites, not just unit tests: replay a fixed set of recorded interviews through the interviewer and judge and diff the scores and behavior. Version the interviewer model, judge model, rubric, and prompts together so you can roll any of them back independently.

Development

Use synthetic candidates and recorded audio fixtures so you can test the loop without live GPUs. A local mode should stub ASR, LLM, and TTS with deterministic fakes, which also makes the turn loop and state machine unit-testable.

Prevention

• Size from peak concurrent sessions and active generations, not user count.
• Budget and continuously measure voice-to-voice latency per stage.
• Combine VAD with semantic turn detection and tune endpointing.
• Model every session as a checkpointed, reconnectable actor.
• Split a fast interviewer model from a thorough async judge.
• Ground scoring in a versioned rubric with evidence and calibration.
• Isolate code execution in per-run microVMs with hard limits.
• Scale GPUs predictively with warm pools and an admission queue.
• Degrade to text on model or provider failure; never drop a session.
• Store transcripts, keep audio opt-in and short-lived, and audit for bias.

Trust, Privacy, and Fairness

This is not a footnote; it is a design constraint. Voice recordings are biometric data under regulations such as GDPR and Illinois BIPA, which means explicit consent, data minimization, retention limits, and deletion rights. If scores ever influence real hiring, you may fall under automated employment decision tool law such as New York City Local Law 144, the EU AI Act's high-risk employment category, and general anti-discrimination rules, which can require published bias audits.

Build for it: collect only what you need, default to transcripts over audio, keep regional data residency, log score evidence for explainability, and run periodic fairness audits across demographic groups. A platform that cannot explain or defend its scores is a liability no matter how good the latency is.

How to Present This in 45 Minutes

1. Minutes 0–5: clarify interview types, scale, and whether scores feed real hiring; derive peak concurrency out loud.
2. Minutes 5–12: draw the three planes and the stateful session model.
3. Minutes 12–22: walk the voice loop and defend the latency budget, especially endpointing and streaming.
4. Minutes 22–30: cover GPU serving, the idle-session insight, and cost per minute.
5. Minutes 30–38: cover failover, reconnection, and graceful degradation.
6. Minutes 38–43: cover async rubric scoring, calibration, and the code sandbox.
7. Minutes 43–45: raise privacy and bias as first-class constraints.

If time is short, protect three things: the latency budget, the stateful session with checkpointing, and grounded scoring. Those are what separate a real design from a diagram.

Key Takeaways

• The system is defined by long, stateful, real-time sessions, so size for peak concurrency and design for reconnection, not requests.
• The voice loop is the product. Budget voice-to-voice latency to ~1 second and treat endpointing and streaming as the primary knobs.
• Sessions are mostly idle, so GPU cost tracks active generations; continuous batching and warm pools make a million users affordable.
• Use the right model for each job: fast interviewer, thorough async judge.
• Make scoring grounded and calibrated, with rubric anchors, cited evidence, low temperature, sampling, and a human gold set.
• Degrade, do not drop. Text fallback and model failover keep interviews alive.
• Privacy and fairness are architecture, because voice is biometric and automated scoring can be regulated.

Suggested Internal Links

• system design interview framework
• how to evaluate AI agents and models
• production LLM observability with OpenTelemetry
• designing a production RAG system
• how to design a multi-agent system
• context engineering interview guide
• WebRTC vs WebSocket for real-time apps
• GPU inference serving with continuous batching
• LLM-as-a-judge evaluation patterns

Official References

• W3C WebRTC specification
• Opus audio codec, RFC 6716
• OpenAI Realtime API guide
• vLLM documentation
• NVIDIA Triton Inference Server
• Firecracker microVM
• gVisor container sandbox
• Kubernetes Horizontal Pod Autoscaler
• NYC DCWP Automated Employment Decision Tools (Local Law 144)
• EU Artificial Intelligence Act

FAQs

How do you design an AI mock interview platform for one million users?

Treat every interview as a stateful, long-lived session, not a stateless request. Put a real-time voice loop (streaming speech-to-text, a fast interviewer LLM, streaming text-to-speech) behind a media edge, run models on GPUs with continuous batching, checkpoint transcripts for failover, and score asynchronously with a rubric-based LLM judge. Size capacity from peak concurrent sessions, which for one million monthly users is on the order of ten thousand, not one million.

What latency does a real-time AI interviewer need?

Aim for voice-to-voice latency near one second, ideally under 1.2 seconds, so turn-taking feels natural. Budget every stage: capture and network (~50 ms), end-of-turn detection (~250 ms), final transcription (~150 ms), LLM first token (~250 ms), first speech chunk (~150 ms), and downlink (~80 ms). End-of-turn detection is the dominant tunable delay; semantic turn detection lets you shorten it without cutting candidates off.

Should you build the voice stack or use a realtime speech API?

Use a managed speech-to-speech API to launch quickly or when premium sub-500 ms latency is the priority. Build a composed speech-to-text to LLM to text-to-speech pipeline when you need turn-by-turn transcripts for scoring, full control of the interviewer model, and the lowest unit cost at high volume. Many teams run a hybrid: managed transcription and speech synthesis with their own LLM.

How many GPUs does a 1M-user AI interview platform need?

Fewer than the user count suggests, because sessions are mostly idle while candidates think and talk. GPU demand tracks active generations, not concurrent sessions. At roughly ten thousand concurrent interviews, only a couple thousand turns are generating at any instant, so with continuous batching you need tens of GPUs per model tier plus redundancy, not thousands. Confirm the exact number with load tests, since it depends on model size and token rate.

How do you make AI interview scoring consistent and fair?

Score asynchronously with an explicit rubric that has anchored 1-to-5 descriptions per dimension. Require the judge to cite transcript evidence before assigning each score, run at temperature zero, sample several times and take the median, and calibrate against a human-labeled gold set. Keep the interviewer and evaluator models separate, version prompts and rubrics, and audit score distributions across demographic groups for bias.

How do you keep a live interview session alive when a server fails?

Hold session state in a worker but continuously checkpoint the transcript and turn state to a durable append-only store. Track presence in a session registry. If the transport drops, move the session to a reconnecting state with a grace window; the client reconnects, is routed back, and the worker rehydrates from the last checkpoint. If the grace window expires, still score the partial transcript rather than discarding the interview.

How do you run candidate code safely at scale?

Execute each submission in a per-execution microVM (Firecracker) or a gVisor-sandboxed container that is destroyed afterward. Disable networking, mount a read-only root filesystem with a small writable tmpfs, drop all Linux capabilities, apply a seccomp profile, run as an unprivileged user, and enforce hard CPU, memory, process-count, and wall-clock limits. Never reuse a sandbox across candidates.

Is an AI mock interview platform subject to hiring-bias laws?

Pure practice tools are lower risk, but the moment scores influence real hiring, automated employment decision tool laws can apply, such as New York City Local Law 144, plus the EU AI Act's high-risk employment category and general anti-discrimination rules. Voice recordings are also biometric data under regulations like GDPR and BIPA. Design for consent, data minimization, retention limits, and periodic bias audits from the start.