Design ChatGPT (LLM Serving)
Serve a conversational LLM to millions with streaming tokens, context management, and GPU efficiency.
Overview
A production LLM chat product serves generated tokens from large models to millions of concurrent users, streams responses in real time, manages conversation context and memory, and does it under brutal GPU-cost constraints. The interview is about the inference-serving layer, not training.
Requirements
Functional
- Multi-turn conversations with streamed, token-by-token responses.
- Manage context windows (up to 100K+ tokens) and conversation memory.
- Authentication, per-user rate limits, and usage/billing.
- Safety moderation on inputs and outputs.
Non-functional
- Low time-to-first-token (< 500 ms) then a steady token stream.
- High availability and graceful degradation under GPU saturation.
- Cost-efficient GPU utilization; the model is the expensive resource.
Back-of-the-envelope
The numbers that justify the architecture.
| Concurrent users | 1M+ | |
| Tokens generated / day | trillions | |
| GPU cost | dominant | inference GPUs dwarf all other infra |
| TTFT target | < 500 ms |
Reference architecture
A streaming API gateway routes chats to an inference service that batches requests across GPUs; context is assembled from a conversation store, with moderation and caching around the model.
Deep dives
Streaming and time-to-first-token
LLMs generate autoregressively — one token at a time. Don't wait for the full response; stream tokens to the client over SSE/WebSocket as they're produced. The metric users feel is time-to-first-token (TTFT); keep it low by minimizing queue wait and prompt-processing (prefill) time. After the first token, throughput is tokens/sec (decode). Splitting prefill vs decode scheduling is a real optimization.
GPU efficiency: batching and the KV cache
GPUs are the cost. Serving one request per GPU wastes it. Continuous (in-flight) batching packs many users' generations into one batch and swaps sequences in/out as they finish, keeping GPUs saturated. The KV cache stores attention keys/values for the tokens generated so far so each new token is cheap — but it consumes GPU memory proportional to context length × concurrency, which is often the real capacity limit (PagedAttention/vLLM manage it). These two ideas are the heart of an LLM-serving answer.
Context, memory, and RAG
Each turn must fit in the context window. The context service assembles the prompt: system prompt + recent history + retrieved memory. For long conversations, summarize or truncate older turns; for external knowledge, retrieve relevant chunks from a vector DB (RAG) and inject them. Longer context = more KV-cache memory and higher cost, so context management is directly a cost lever.
Cost, caching, and degradation
Cache identical/similar prompts (and cache system-prompt prefixes — prefix KV reuse). Route easy queries to smaller/cheaper models and hard ones to the flagship (model cascading). Under GPU saturation, degrade gracefully: queue with a visible wait, shed to a smaller model, or rate-limit — never hard-fail everyone. Moderation runs on both input and output as a safety gate.
Key trade-offs
For each decision: the two options, and when to pick which.
Bottlenecks & follow-ups
Where it breaks under load — and what an interviewer will probe.
- ▲GPU memory (KV cache) caps concurrency → PagedAttention, bounded context.
- ▲GPU utilization → continuous batching, not per-request.
- ▲Tail latency under load → queue with backpressure + model fallback.
What a strong answer sounds like
- ✓Say 'continuous batching + KV cache' early — it's the signal you understand LLM serving.
- ✓Frame every decision around GPU cost; it's the dominant constraint.
- ✓Distinguish TTFT (prefill) from throughput (decode) — shows real depth.