Systems labAdvanced145 min estimateOriginal publication

Design a Multi-Tenant AI Gateway

A gateway earns its place only when it centralizes cross-cutting control without becoming a universal bottleneck or hiding product-specific quality decisions.

Authorship: InterviewsVector
Published / updated: 2026-08-11 / 2026-08-11
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

Place a stateless gateway on the model-call boundary with authenticated tenant context, policy evaluation, capability-based routing, provider adapters, bounded retries, rate and cost budgets, trace emission, and deterministic validation. Keep prompts, product evals, and business fallbacks owned by product teams. Separate the request data plane from configuration, budgets, model inventory, and rollout control planes.

Why this matters

Without a shared boundary, every product reimplements credentials, provider quirks, budgets, safety, telemetry, and failover. With an overreaching gateway, one team owns every prompt and becomes the bottleneck. The architecture problem is choosing the boundary.

You will be able to

Separate gateway data plane, control plane, and evidence plane.
Design tenant isolation across identity, quotas, data, logs, caches, and providers.
Route by capabilities and policy rather than brittle provider names.
Calculate token throughput, concurrency, and cost-envelope assumptions.
Define fallback and retry behavior that does not multiply load or change semantics silently.

Prerequisite contract

— HTTP gateways, rate limiting, and distributed tracing
— LLM inference and provider APIs
LLM gateway design →

Your Vector Loop for this lab

01
Model
Define the gateway as a policy boundary, not a pass-through proxy.
02
Derive
Translate traffic shape into tokens, concurrency, cost, and dependency budgets.
03
Build
Implement a capability contract and deterministic capacity model.
04
Stress
Inject provider, tenant, budget, cache, and validation failures.
05
Operate
Add SLOs, rollouts, evidence, regional policy, and graceful degradation.
06
Defend
Defend what the platform owns and what remains with product teams.

Draw three planes before choosing products

Gateway boundary and ownership

Tenant-authenticated requests cross policy and routing into provider adapters, while configuration and evaluation evidence remain separate planes.

The gateway is a policy and evidence boundary, not a thin provider proxy. Product teams still own prompts, task evaluation, and user experience.

Plane	Owns	Examples
Data	per-request enforcement and routing	auth context, validation, budgets, retry, streaming
Control	versioned policy and rollout state	model catalog, tenant limits, regions, adapter versions
Evidence	quality and operating records	traces, eval results, cost, safety events, release decisions

Route capabilities, not vendor-shaped payloads

gateway-contract.ts

1type GatewayRequest = {
2  tenantId: string
3  workload: "interactive" | "batch" | "agent"
4  capabilities: { vision?: boolean; tools?: boolean; jsonSchema?: boolean }
5  limits: { maxInputTokens: number; maxOutputTokens: number; maxCostUsd: number }
6  policy: { dataRegion: string; allowTrainingRetention: false }
7  traceId: string
8}
9 
10type Route = { provider: string; model: string; reason: string; fallback?: string }

Verify: Run npm run typecheck to validate the contract with the application type graph.

Adapters translate the internal contract to provider APIs and normalize finish reasons, usage, errors, tool calls, and streaming events back. Keep provider-only features behind explicit capabilities so switching a route cannot silently drop schema enforcement, regional policy, or tool behavior.

Tenant isolation is more than a rate-limit key

Authenticate identity upstream and bind tenant context cryptographically; never accept a free-form tenant field from the body.
Scope provider credentials, encryption, caches, logs, traces, and budgets by tenant and environment.
Enforce per-tenant and global concurrency so one noisy tenant cannot consume every downstream slot.
Keep prompt and response capture opt-in with retention, redaction, and regional controls.
Include tenant and policy version in cache keys; semantic caches can leak data or unsafe equivalence across contexts.
Make emergency disable, model deny-list, and budget stop available without redeploying every client.

Calculate demand in tokens and occupied time

token demand/s = requests/s × (mean input tokens + mean output tokens)

This is a first approximation. Capacity must be split by workload, model, prompt length, output length, cache behavior, and burst shape.

concurrency ≈ arrival rate × mean occupied seconds

Little's Law gives an initial concurrency estimate. Streaming connections, tool pauses, queues, and provider limits need separate treatment.

gateway-capacity.ts

1type Workload = {
2  requestsPerSecond: number
3  meanInputTokens: number
4  meanOutputTokens: number
5  meanOccupiedSeconds: number
6  burstMultiplier: number
7}
8 
9function tokenDemandPerSecond(w: Workload) {
10  return w.requestsPerSecond * (w.meanInputTokens + w.meanOutputTokens) * w.burstMultiplier
11}
12 
13function expectedConcurrency(w: Workload) {
14  return w.requestsPerSecond * w.meanOccupiedSeconds * w.burstMultiplier
15}
16 
17const workload: Workload = {
18  requestsPerSecond: 10,
19  meanInputTokens: 1200,
20  meanOutputTokens: 400,
21  meanOccupiedSeconds: 0.75,
22  burstMultiplier: 2.4,
23}
24 
25console.log(`${tokenDemandPerSecond(workload).toLocaleString("en-US")} tokens/s`)
26console.log(`${expectedConcurrency(workload).toFixed(1)} concurrent requests`)

Expected output

38,400 tokens/s
18.0 concurrent requests

Verify: Run node --experimental-strip-types --test courses/ai-engineering/reference-impl/gateway/gateway-capacity.test.ts.

Limit	Scope	Failure behavior
request rate	tenant + global	429 with retry guidance
concurrency	workload + model + tenant	queue or shed
token rate	tenant + provider	route, delay, or reject
spend	request + daily + monthly	deny before commitment
context	model capability	truncate only by explicit policy

Fallback must preserve a declared product contract

01FilterRemove routes that violate capability, region, data, safety, or tenant policy.
02ScoreUse measured quality, latency, availability, and cost for this workload and slice.
03AdmitCheck current budget, concurrency, provider limits, and expected token growth.
04ExecuteApply one bounded retry only when semantics and idempotency allow it.
05ValidateEnforce deterministic output schema and product safety boundary.
06DegradeUse an approved smaller model, cached answer, queued workflow, or refusal—not an invisible quality change.

Observe the user journey and the dependency

Signal	User-facing view	Dependency view
availability	valid response or declared degradation	provider success by route
latency	time to first token and completion	queue, connect, prefill, decode
quality	task eval and user outcome	model/version slice
cost	cost per successful task	tokens and provider charges
safety	policy-compliant outcome	blocked category and control

OpenTelemetry's generative-AI semantic conventions provide shared vocabulary for model and agent operations, but trace capture is a privacy decision. Prefer identifiers, counts, model metadata, timings, and redacted evidence by default; capture raw content only with explicit need, controls, and retention.

Ship the gateway as a platform, not a monopoly

Publish a capability contract, SDK, migration guide, SLO, and ownership map.
Canary adapter and policy changes using shadowed, redacted requests and workload-specific evals.
Let product teams pin routes temporarily during incidents or investigations with audited expiry.
Keep a direct emergency path only if it retains authentication, budget, and audit controls.
Charge back or show back cost per tenant and workload without turning raw token count into a quality target.
Review whether the gateway still reduces duplicated risk; delete features that only centralize preference.

Operate at three altitudes

Production lens

— Use one canonical retry budget across clients, gateway, adapters, and SDKs.
— Version policies, routes, adapters, and model capabilities; emit every resolved version in traces.
— Protect content capture with tenant controls, redaction, encryption, retention, and access auditing.
— Test provider failure, brownouts, malformed streams, rate limits, budget exhaustion, and region loss.

Staff lens

— Keep cross-cutting safety and reliability controls centralized while product quality ownership stays close to users.
— Use workload-specific eval evidence in routing; avoid a universal model leaderboard.
— Design an exit path so the gateway does not become an irreversible vendor abstraction or organizational bottleneck.

Interview defense

Design a multi-tenant LLM gateway for an organization.

I would create a stateless data plane for authenticated tenant context, policy, capability routing, budgets, bounded retry, streaming normalization, validation, and traces; a versioned control plane for models, adapters, limits, region and rollout policy; and an evidence plane for quality, cost, safety, and release decisions. I would isolate credentials, quotas, caches, logs, and content capture per tenant, route only among policy-compatible options, and make degradation explicit.

Expect the interviewer to press on

— What should remain owned by product teams?
— How do you prevent a retry storm?
— How would you route by quality without leaking evaluation data?
— What is the rollback unit for a policy change?

Misconceptions to remove

“A gateway makes providers interchangeable.”

It can normalize a useful common contract, but model capabilities, semantics, safety behavior, and operating characteristics remain different.

“Centralizing prompts creates consistency.”

It often removes product ownership and slows learning; centralize controls and reusable primitives, not every domain decision.

“Provider failover guarantees reliability.”

Fallback may violate capability, policy, quality, or data-region requirements and can amplify load during correlated incidents.

Check your model

1. Why separate the control plane from the request data plane?

Configuration, rollout, inventory, and policy evolve differently from low-latency request execution; separation reduces blast radius and supports versioned rollback.

2. Why is cost per token an incomplete platform metric?

The product needs cost per successful task or outcome; a cheap call that fails quality and triggers retries can be more expensive.

Prove the mechanism

Write an architecture decision record for gateway ownership. Include capability contract, tenant isolation, retry ownership, SLOs, data retention, and a two-provider failure exercise.

Add a production constraint

Implement a simulator that routes workload traces under quality, latency, budget, region, and concurrency constraints, then compare graceful-degradation policies.

Artifact: Gateway architecture and capacity model

courses/ai-engineering/reference-impl/gateway/gateway-capacity.ts

Download reference implementation

Primary references and next links

References

1. Service Level Objectives
Google SRE. Primary practical framework for user-centered service objectives.
2. Generative AI semantic conventions
OpenTelemetry. Official semantic-convention specification for generative-AI telemetry.
3. OWASP Top 10 for LLM Applications 2025
OWASP GenAI Security Project. Current application-security risk catalog and mitigations.

Continue through the graph

LLM gateway system design →
Rehearse the canonical architecture answer.
Cost-aware model routing →
Deepen the routing policy.
Shared AI platform strategy →
Connect technical boundaries to organizational ownership.
Architecture Judgment Index →
Practice adjacent Staff and Architect decisions.

Glossary: AI gateway · data plane · control plane · multi-tenancy · SLO · rate limit · graceful degradation