Build labAdvanced110 min estimateOriginal publication

Load-Test an AI Service

Turn an AI workload model into reproducible pressure without letting averages, generator saturation, retries, cache warmth, or dropped quality checks manufacture a passing result.

Authorship: InterviewsVector
Published / updated: 2026-09-26 / 2026-09-26
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary sources; they do not claim hardware benchmarks, distributed execution, production deployment, capacity certification, or universal model quality.

The decision in one pass

Load-test the service users actually create, not a single prompt at a fixed concurrency. Version a workload model covering prompt and output shapes, arrival patterns, concurrency, tenant or priority mix, cold and warm cache states, cancellations, retries, streaming behavior, and quality sampling. Separate offered load from achieved throughput; prove the generator retains headroom so client saturation is not mistaken for server capacity. Measure time to first token and end-to-end tails alongside errors, cancellations that continue consuming work, retry amplification, useful throughput, resource pressure, and quality. Require the complete Cartesian slice set and a minimum sample per slice. Apply latency and error, cancellation-leak, retry, and quality gates to every required slice before reporting aggregates, because a large easy slice can numerically hide a catastrophic small one. Use bounded ramp, steady, spike, overload, and recovery phases with explicit stop conditions. The reference audit checks invented aggregate evidence only: it generates no traffic and is neither a production capacity result nor an MLPerf submission.

Why this matters

AI request cost varies dramatically with prompt length, generated tokens, model revision, batching, cache state, tool or retrieval behavior, and cancellation timing. A test can look healthy while the generator is saturated, retries inflate offered work, cancelled generations keep decoding, long requests starve, cache warmth is unrealistic, or quality collapses under a fallback path. Representative slices and causal telemetry make capacity and failure behavior explainable instead of reducing the service to one throughput number.

You will be able to

Build a versioned workload model from sequence, arrival, concurrency, cache, cancellation, retry, tenant, and quality dimensions.
Distinguish offered load, admitted work, completed useful work, achieved throughput, concurrency, and latency.
Design bounded steady, ramp, spike, overload, and recovery phases while preserving generator headroom and reproducibility.
Gate every required slice independently so imbalanced sample volumes cannot hide a catastrophic cohort.
Defend the difference between a local synthetic audit, production capacity qualification, and a standardized benchmark result.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Map production arrival and sequence distributions, concurrency, tenants, priorities, cache states, cancellations, retries, streaming, dependencies, quality rubric, service revision, resource limits, and safety boundaries.
02
Derive
Derive offered-load phases, per-slice sample floors, latency and reliability limits, quality sampling, generator-headroom proof, overload stops, recovery criteria, and the dimensions that must never be averaged away.
03
Build
Build a reproducible harness and a deterministic local auditor over invented digest-bound slice evidence; separate traffic generation, observation, and release decision authority.
04
Stress
Inject long outputs, bursty arrivals, cold caches, cancellation storms, retry feedback, generator saturation, dependency slowdown, telemetry gaps, quality loss, noisy neighbors, overload, and slow recovery.
05
Operate
Observe arrivals, admissions, queues, active sequences, prefill and decode work, TTFT and end-to-end tails, completions, cancellations, leaked work, retries, cache state, quality, resources, generator headroom, and recovery.
06
Defend
Defend representativeness, per-slice gates, client/server separation, stopping safety, quality preservation, reproducibility, and the limited claim the evidence can support.

Represent the joint workload, not one average request

Start from privacy-preserving production distributions or an explicitly hypothetical target workload. Preserve correlations that determine cost: long prompts may pair with short outputs, particular tenants may burst, cache hits may concentrate by route, and cancellations may arrive late in decode. Version the service revision, tokenizer, generation policy, prompt construction, tool or retrieval behavior, and workload manifest. Synthetic text may protect data, but its token and control-flow behavior still has to match the dimension being tested.

Dimension	Why it changes the system	Evidence to retain
prompt/output shape	prefill, KV growth, decode duration, batching compatibility	joint token buckets and output-stop reason
arrival/concurrency	queueing, batch formation, admission, overload	timestamped offered and admitted work
cache state	load path, prefix/KV reuse, memory pressure	cold/warm policy and hit identity
cancellation	wasted decode, cleanup, stream termination	cancel time, stopped work, leaked work
retry	feedback amplification and duplicate work	original request identity and every attempt
quality	fallback or overload can silently degrade answers	rubric version, sampled completion, decision

Separate offered load from achieved throughput

A closed-loop client sends new work after completions, so server slowdown reduces the offered rate and can conceal overload. An open-loop schedule targets arrivals independently, exposing queues but requiring bounded backpressure and a generator that can keep time. Neither label is sufficient by itself: record scheduled, attempted, admitted, cancelled, failed, and completed work, plus the generator's CPU, network, timer lag, and connection limits. If the generator lacks declared headroom, the result is incomplete.

concurrency ≈ arrival_rate × mean_time_in_system

Little's Law is a consistency check for a stable observation window, not permission to replace tail distributions or unstable overload behavior with one mean.

01Warm deliberatelySeparate artifact load, kernel or graph warmup, connection setup, and declared cache warmup from measured cold and warm slices.
02Run bounded phasesUse steady baselines, controlled ramps, representative spikes, a safe overload boundary, and a recovery window with explicit time and resource stops.
03Preserve request identityJoin original requests, retries, cancellations, responses, quality samples, server spans, and resource signals without counting attempts as independent demand.
04Prove generator headroomMonitor scheduler lag, client resources, sockets, network, and send rate so a flat throughput curve can be attributed to the service rather than the driver.

Gate tails and quality per required slice

Aggregates remain useful for cost and fleet reporting, but they are unsafe release gates when slice volumes differ. A hundred thousand easy warm-cache requests can hide a thousand-request cold long-sequence failure. Require exact coverage, minimum volume, and complete telemetry for every declared sequence × traffic × cache slice. Then apply TTFT, end-to-end, error, cancellation-leak, retry-amplification, and quality limits to each slice. Quality checks must describe completed responses, so their count cannot exceed completions; a slice with no quality evidence fails the quality gate instead of producing a division artifact.

Signal	Denominator or context	Failure it exposes
p95 TTFT	successful response starts within one slice	queueing or prefill delay
p99 end-to-end	complete request duration within one slice	long decode and tail starvation
error rate	all requests in the slice	admission, execution, or dependency failure
cancellation leak	cancelled requests in the slice	work continuing after callers leave
retry amplification	attempts relative to original requests in the slice	positive feedback under distress
quality pass rate	checked completed responses in the slice	degraded output hidden by availability

Audit a complete synthetic evidence matrix

The reference artifact consumes invented summaries; it does not open connections or generate prompts. Frozen records bind the workload digest, service revision, deterministic seed, scope, and contract. Validation rejects missing or duplicate slices, mixed runs, outcome counts that do not partition requests, quality checks beyond completed responses, non-finite or wrong-type numbers, string subclasses, forged records, tampered digests, stale scopes, mutation, and incomplete telemetry. The report retains aggregate display metrics while the verdict uses per-slice gates.

courses/ai-engineering/reference-impl/ai_load_test/load_test_gate.py

1def audit_load_test(
2    contract: LoadTestContract, evidence: LoadTestEvidence
3) -> LoadTestReport:
4    contract = validate_record(contract, LoadTestContract)
5    evidence = validate_record(evidence, LoadTestEvidence)
6    if evidence.scope != contract.scope or evidence.contract_content_id != contract.content_id:
7        raise ValueError("evidence belongs to another load-test contract")
8 
9    # Require the complete slice matrix and minimum evidence per slice.
10    # Apply latency, error, cancellation, retry, and quality gates per slice.
11    # Aggregate rates remain descriptive and cannot hide a failing cohort.

Expected output

example=illustrative_only
decision=PASS_LOAD_TEST
slices=18;requests=18000
worst_ttft_ms=440.000;worst_e2e_ms=2100.000
error_rate=0.005;cancel_leak_rate=0.000
retry_amplification=1.020;quality_pass_rate=0.980
claim=LOCAL_SYNTHETIC_GATE_NOT_CAPACITY_CERTIFICATION

Verify: python3 -m unittest discover -s courses/ai-engineering/reference-impl/ai_load_test -p 'test_*.py' -v

Gate an AI load-test result

Inspect invented sequence-shape, traffic, cache, cancellation, latency, retry, and quality evidence. Predict pass, hold, or fail before revealing the declared local gate; no traffic is generated.

Gate a capacity claim with representative load

Inspect sequence shapes, concurrency, cancellations, tail latency, and recovery as one versioned workload contract. Predict what the evidence actually authorizes.

Load-test record

The test uses one convenient request shapeMeasured throughput is high, but the generator excludes long prompts and long decodes in the serving forecast.Cancelled streams retain serving resourcesThe latency target passes until client churn accumulates orphaned KV blocks and queue entries.Averages pass while the tail violates SLOThe target concurrency is sustained, but interactive users cross the declared time-to-first-token limit.Representative load stays inside every gateThe forecast mix, churn, warmup, steady state, and recovery all satisfy the declared envelope.

Traffic, services, environments, and measurements are illustrative. This lab does not generate traffic or certify production capacity.

Capacity-gate evidence
Signal	Operating constraint	Observed evidence
Prompt distribution	Match the versioned forecast	Fixed at 128 tokens
Forecast prompt p95	Represent in test mix	4,096 tokens
Decode distribution	Match the versioned forecast	Fixed at 64 tokens
Reported throughput	Valid only for declared mix	11,400 tokens/s

Illustrative measurements only. The recommended action stays hidden until you check a prediction.

What is the defensible disposition for this load test?

Rebuild the workload mix before testing againFix cancellation and recovery behaviorHold launch and reduce the capacity claimAccept the bounded capacity result

Choose an action before checking the operating contract.

Test overload without creating an incident

Overload is a feedback experiment, not a contest to maximize requests. Cap duration, cost, concurrency, queued work, retries, and resource pressure; isolate the environment and obtain production authorization if real systems are involved. Increase one controlled dimension at a time until a stop condition, then observe recovery. The service should shed work predictably, preserve high-priority capacity and quality policy, honor cancellations, bound retries, and return queues and tails to baseline without operator guesswork.

Disable unbounded automatic retries in the generator; record every attempt against the original request.
Include cancellation storms and verify decode, memory, queue slots, and downstream calls stop within a declared bound.
Slow or fail dependencies so the test captures timeout alignment, bulkheads, circuit breaking, and retry amplification.
Exercise cache-cold restart and cache eviction rather than reporting only a permanently warm steady state.
Measure recovery time, queue drain, error decay, quality restoration, and residual resource leaks after offered load falls.
Abort on safety, spend, saturation, data, or blast-radius limits even if the planned phase has not completed.

Publish a capacity claim with its boundary

A defensible result names the service and model revision, hardware and topology, runtime and kernels, autoscaling state, workload manifest, seed, token distributions, generation policy, cache conditions, phase schedule, generator version and headroom, dependencies, samples, exclusions, gate contract, quality rubric, raw evidence location, and time. Capacity is an operating envelope, not a permanent scalar. Re-run after changes to models, quantization, prompts, scheduling, hardware, dependencies, routing, or policy.

01State the supported envelopeName the arrival and sequence distribution, concurrency, cache, quality, latency, error, and resource boundaries rather than advertising an unqualified maximum.
02Retain causal evidenceKeep request identities, server traces, scheduler state, resource telemetry, generator health, quality samples, and configuration manifests long enough to explain the result.
03Separate local and standardized claimsA local workload qualification answers your declared product question. MLPerf results follow their published scenarios, rules, systems, and submission process and should not be implied by a custom harness.
04Turn the envelope into controlsFeed admission, routing, autoscaling, budgets, alerting, and rollout policy from the qualified boundary, then detect when production moves outside it.

Operate at three altitudes

Production lens

— Join offered, admitted, completed, cancelled, retried, and quality-checked work with queue, batch, cache, resource, dependency, and generator signals by revision and workload slice.
— Exercise representative steady, burst, overload, cancellation, retry, cold-cache, dependency, noisy-neighbor, and recovery conditions under explicit blast-radius and spend stops.
— Publish per-slice gates and the complete evidence manifest; use aggregates for reporting without allowing high-volume easy traffic to erase a required failure.

Staff lens

— Own load testing as a product-workload, distributed-systems, measurement, safety, quality, capacity, and cost discipline rather than a single requests-per-second script.
— Reject capacity claims that omit workload correlations, generator headroom, per-slice tails, cancellation and retry behavior, quality evidence, recovery, revision identity, or claim boundaries.

Interview defense

How would you load-test a streaming LLM service whose production traffic mixes short chats, long documents, cancellations, and retries?

I would derive a versioned joint workload model from privacy-safe evidence: prompt and output buckets, arrival patterns, concurrency, tenant and priority mix, cold and warm cache, streaming, cancellation timing, retry policy, and quality samples. I would bind it to the exact model, tokenizer, generation, scheduler, hardware, and service revision. A generator with measured headroom would run bounded steady, ramp, spike, overload, and recovery phases while recording offered, admitted, completed, cancelled, and retried work. I would measure TTFT and end-to-end tails, errors, cancellation leaks, retry amplification, useful throughput, queues, resources, dependencies, and quality. Coverage, sample floor, telemetry, and every performance and quality gate would apply per required slice so large short requests cannot hide a long-document failure. I would stop on explicit safety and cost limits, prove recovery, retain causal evidence, and describe the result as a local operating envelope rather than a universal or standardized benchmark.

Expect the interviewer to press on

— How can a closed-loop generator hide overload?
— Why must quality checks be bounded by completed responses?
— What evidence proves the load generator is not the bottleneck?

Misconceptions to remove

“One fixed prompt and concurrency level measures service capacity.”

It measures one synthetic point. AI cost and behavior depend on correlated sequence, arrival, cache, cancellation, retry, tenant, dependency, and quality conditions.

“A good fleet-average error rate means every required workload is safe.”

Large easy slices can dilute catastrophic smaller cohorts. Gate every declared slice independently and retain aggregates only as descriptive summaries.

“A custom synthetic pass is equivalent to a standardized benchmark result.”

It supports only its declared local workload and environment. Standardized results require the published benchmark's scenarios, rules, system description, and submission process.

Check your model

1. Why record offered load separately from completed throughput?

The server may queue, reject, cancel, or fail work, and a saturated client may stop offering it. Completed throughput alone cannot identify the bottleneck or overload response.

2. What is a cancellation leak?

Work that continues consuming compute, memory, queue, or dependency capacity after the caller's cancellation should have stopped it under the declared bound.

3. Why can zero quality checks fail a slice even when availability looks healthy?

Without evaluated completed responses, the test has no evidence that overload, routing, fallback, truncation, or other behavior preserved the declared quality floor.

Prove the mechanism

Extend the local audit with a tenant-and-priority dimension and separate admission and completion counts. Add an imbalanced high-volume slice that would hide a low-volume quality and cancellation failure under aggregate gates.

Add a production constraint

Design a safe distributed load campaign across two serving regions and a constrained dependency. Specify the workload manifest, generator synchronization and headroom, open- and closed-loop phases, cancellation and retry identity, quality sampling, overload stops, recovery proof, evidence retention, operating-envelope decision, and production rollback plan.

Artifact: AI load-test plan