Systems labAdvanced120 min estimateOriginal publication

The KV Cache Capacity Plan

Model weights are the static bill. KV state is the request-shaped bill that grows during decode, competes with batching, and must be reserved under a named model, allocator, scheduler, and capacity contract.

Authorship: InterviewsVector
Published / updated: 2026-08-11 / 2026-08-25
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector material. Code examples are covered by repository tests and primary references are recorded. No named human reviewer is claimed.

The decision in one pass

For a decoder transformer, raw KV-cache bytes per sequence are cached tokens × layers × two tensors × KV heads × head dimension × bytes per element. Grouped- or multi-query attention reduces KV heads; cache precision changes element width. Operational reservation then rounds prompt-plus-maximum-output tokens to allocator blocks, multiplies by sequences, and applies a declared overhead convention. Available cache capacity is device bytes minus weights, runtime workspaces, graph captures, and failure headroom—not the headline device size. Bind model and architecture digests, cache precision, allocator, scheduler, reservation policy, device pool, request shapes, and reported capacity. Admit only from revalidated content-addressed evidence, and treat the result as capacity arithmetic requiring load validation rather than an SLO guarantee.

Why this matters

A service can pass single-request latency tests and still fail at modest concurrency because long prompts and uncertain outputs reserve the cache needed for continuous batching. A calculator with mutable inputs or no scheduler and allocator identity makes a precise-looking number that cannot support admission control or incident replay.

You will be able to

Derive cache bytes from decoder key/value tensor shapes.
Distinguish MHA, GQA, and MQA cache width from query-head compute.
Separate raw cache, block rounding, allocator overhead, non-cache reservations, and failure headroom.
Bind model, cache precision, allocator, scheduler, capacity, request-shape, and reservation evidence.
Stress booleans, invalid precision, unsafe counts, overflow, stale scope, and tampered records.
Translate a content-addressed plan into admission, load testing, SLOs, and rollback.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Name model, cache tensor shape, precision, allocator blocks, scheduler, capacity pool, headroom, and request reservation policy.
02
Derive
Derive exact per-token bytes, block-rounded reserved tokens, basis-point overhead, and remaining capacity with integer arithmetic.
03
Build
Capture immutable request-shape evidence and produce a content-addressed FIT or EXCEEDS_CAPACITY plan.
04
Stress
Vary GQA ratio, precision, prompt and output tails, concurrency, block size, overhead, cancellation, scope, and arithmetic limits.
05
Operate
Compare predicted occupancy with allocator telemetry, load tests, latency slices, eviction, release lag, and admission outcomes.
06
Defend
Defend assumptions and uncertainty while refusing to turn capacity arithmetic into a latency or throughput guarantee.

Cache prior keys and values as request state

During autoregressive decode, each new query attends to prior positions. Retaining the key and value projections at every decoder layer avoids recomputing those projections for the prefix on every step. The cache is mutable per-sequence state: it grows as tokens are generated and should be released promptly when a sequence finishes or cancels.

Memory	Lifetime	Capacity treatment
model weights	deployment revision	reserve before cache
runtime workspaces/graphs	kernel or shape dependent	reserve measured headroom
KV cache	active sequence	admit by request-shape reservation
temporary attention buffers	operator execution	benchmark by kernel and shape

Derive raw bytes from the cache tensor

bytes_raw = S × T × L × 2 × Hkv × Dhead × B

S is sequences, T cached tokens per sequence, L decoder layers, 2 accounts for keys and values, Hkv is key/value heads, Dhead is head width, and B is bytes per cached element.

Query-head count still affects attention computation; KV-head count determines the cached projection width in this formula.
Attention variant	KV heads	Raw cache effect
multi-head attention	equals query heads	baseline width
grouped-query attention	between one and query heads	linear reduction by KV-head ratio
multi-query attention	one	minimum KV-head width

Precision must be an explicit cache format, not a free floating-point guess. Packed sub-byte formats need layout and alignment rules; the compatibility helper accepts a bounded set of element widths, while the operational contract binds a cache-precision identity alongside model shape.

Reserve growth, blocks, overhead, and headroom

Talloc = ceil(Treserve / Tblock) × Tblock

The reservation covers prompt plus declared maximum output growth, then rounds to the allocator block. The artifact multiplies exact integer bytes and applies overhead in basis points with ceiling, so admission never benefits from downward rounding.

01Subtract non-cache memoryReserve weights, runtime, graphs, monitoring, and failure headroom before assigning cache capacity.
02Reserve request growthUse prompt tokens plus maximum permitted new tokens or a separately governed probabilistic reservation policy.
03Round by allocator contractAccount for block size and declared overhead; do not use average fragmentation as a universal constant.
04Bind scheduler semanticsContinuous batching, beam count, prefix sharing, preemption, and cancellation change active sequence occupancy.

Grade the reservation before admitting it

Inspect declared model shape, cache capacity, non-cache headroom, allocator overhead, prompt length, output reservation, and concurrency. Predict fit or exceed, then check to reveal the calculation.

Decide whether a KV-cache workload fits its declared envelope

Set context length and concurrency for one fixed grouped-query model. Account for K and V storage, capacity headroom, and non-cache overhead before you classify the workload. The arithmetic remains hidden until check.

Context tokens per request2,048Concurrent requests64

Decision contract

The workload fits only when KV bytes are no greater than capacity after both the safety headroom and fixed runtime overhead are reserved. Equality counts as fit.

Model constants

Layers: 32
KV heads: 8
Head dimension: 128
Element width: 2 bytes

Capacity contract

Physical capacity: 24 GiB
Safety headroom: 15%
Fixed overhead: 3 GiB
Unit basis: 1 GiB = 2³⁰ bytes

KV bytes = 2 × layers × KV heads × head dimension × element bytes × context tokens × requests

Per-token bytes, total KV demand, and remaining safe capacity are withheld until you check a prediction.

This is an illustrative capacity exercise with a fixed formula. It excludes allocator fragmentation, prefix sharing, and implementation-specific metadata.

Does the current workload fit the declared capacity envelope?

Fits, including the equality boundaryExceeds the safe KV-cache budget

Make a prediction, then check it against the current evidence.

Build a content-addressed capacity plan

kv_cache_budget.py

1def build_capacity_plan(
2    contract: CapacityContract, evidence: ReservationEvidence
3) -> CapacityPlan:
4    """Revalidate reservations and return a content-addressed admission plan."""

Expected output

example=illustrative_only
contract_version=kv-capacity-plan-v2
model_content_id=kv-model@sha256:031a251a4cd31746a9c99deb5c4fcd9b6ac2948d0d588e2e72b5c00617b28dea
evidence_content_id=kv-reservation-evidence@sha256:18efdc4968ccebdd278c347ce8e0608ca9206eddfc36d53cd85a439bcb63d1cf
requests=2
sequences=16
reserved_tokens=98304
required_cache_gib=13.800
available_cache_gib=40.000
decision=FIT
plan_content_id=kv-capacity-plan@sha256:453ff9eb7e068f6197f897416a83d65f1715b01017529fc2140a4795200f0c5f
claim=CAPACITY_ARITHMETIC_ONLY_REQUIRES_LOAD_VALIDATION

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/kv_cache

The excerpt is literal artifact source. The upgraded artifact keeps the original ModelShape, cache_bytes, gib, and max_sequences helpers while adding concrete frozen capacity, request, evidence, and plan records. The public boundary revalidates model and architecture identity, cache precision, scheduler, allocator, device pool, capacity, reservations, block rounding, overhead, owners, and content digests.

Stress the long tail and the evidence boundary

Scenario	Expected plan behavior	Operational follow-up
long prompt plus output tail	reserve full declared growth	slice queueing and latency by shape
block boundary + 1 token	round to next block	measure allocator waste
stale scheduler or allocator	reject scope	recapture evidence
capacity reported above contract	reject evidence	reconcile device reservation
duplicate request identity	reject support	fix scheduler logging
aggregate byte overflow	fail closed	reduce bounded planning window

01Replay cancellation lagMeasure how long blocks remain allocated after disconnect; nominal active request count can understate occupancy.
02Replay prefix sharingBind tenant and invalidation semantics before treating shared blocks as saved capacity.
03Replay beam or multi-sequence requestsReserve every live sequence, not merely the external request count.
04Attack record immutabilityRefuse lists, duck types, constructor-bypassed booleans, tampered content IDs, and reused request identities.

Validate the plan against a live workload distribution

Signal	Slice	Decision it informs
allocated and free blocks	model, pool, allocator	admission headroom
prompt/output tokens	tenant, task, percentile	reservation policy
TTFT	prefill tokens and queue	prefill scheduling
inter-token latency	batch occupancy and output length	decode scheduling
eviction/recompute/release lag	stop and cancellation reason	allocator and lifecycle fixes

Load tests must preserve arrival bursts and the joint prompt/output distribution. Compare predicted and observed occupied bytes by shape, include cancellation and timeouts, and report latency percentiles rather than one requests-per-second number. Refit overhead only through a versioned contract with evidence; never tune it silently after an incident.

Defend capacity arithmetic without promising an SLO

A FIT plan means the block-rounded, overhead-adjusted reservations in one content-addressed window do not exceed the declared available cache bytes. It does not prove allocator efficiency, kernel workspace behavior, prefix-cache hit rate, latency, throughput, queue stability, hardware health, or output-quality policy. Those claims need measurements under the intended workload.

Operate at three altitudes

Production lens

— Track allocated/free cache blocks, internal waste, prefix sharing, evictions, recompute, and release latency by model and pool.
— Slice TTFT and inter-token latency by prompt length, reserved output, actual output, sequence count, and batch occupancy.
— Version model architecture, cache precision, allocator, scheduler, reservation policy, device capacity, and non-cache headroom together.
— Compare predicted reservation bytes with observed occupancy under bursty joint request-shape distributions.

Staff lens

— Keep capacity planning, scheduler policy, allocator behavior, and SLO validation as connected but separate evidence surfaces.
— Require request-shaped admission and explicit failure headroom before increasing concurrency.
— Review prefix sharing with tenant isolation, privacy, invalidation, and accounting—not throughput alone.

Interview defense

How would you capacity-plan KV cache for an LLM service?

I would derive tokens × layers × keys-and-values × KV heads × head dimension × bytes per element per sequence, using the actual GQA or MQA shape. I would subtract weights, runtime workspaces, graphs, and failure headroom from device capacity; reserve prompt plus permitted output growth; round to allocator blocks; apply declared overhead; and multiply by live sequences. I would bind model, precision, allocator, scheduler, pool, and request evidence, then validate predicted occupancy and latency against the joint prompt/output distribution. FIT arithmetic is not an SLO guarantee.

Expect the interviewer to press on

— How does GQA change cache size without changing query-head count?
— Why can average context length overstate safe concurrency?
— What must be true before you count prefix sharing as capacity?

Misconceptions to remove

“The accelerator has 80 GiB, so 80 GiB is available to KV cache.”

Weights, runtime workspaces, graphs, monitoring, and failure headroom must be reserved first.

“Paged allocation removes the need for output reservation.”

Paging reduces some fragmentation and enables flexible allocation; active sequences still grow and consume bounded physical memory.

“A FIT estimate proves the service will meet latency SLOs.”

Capacity arithmetic does not model queueing, bandwidth, compute, kernels, scheduler dynamics, or workload bursts without load evidence.

Check your model

1. If KV heads drop from 32 to 8 with all other raw-cache variables fixed, how does size change?

It drops by a factor of four because raw cache width is linear in KV-head count.

2. Why round reserved tokens to allocator blocks before admission?

Physical allocation consumes whole blocks, so using unrounded token counts can admit work that does not actually fit.

3. Why bind scheduler and allocator versions when the model shape is unchanged?

They determine sequence multiplicity, reservation lifecycle, block rounding, sharing, preemption, and release behavior that change usable capacity.

Prove the mechanism

Capacity-plan an 80 GiB pool for two request-shape cohorts under MHA and GQA. Bind all identities, reserve non-cache memory, round blocks, and compare predicted with measured occupancy.

Add a production constraint

Build a discrete-event simulation of bursty arrivals, continuous batching, cancellation, prefix sharing, and output uncertainty. Keep its scheduler and allocator claims separate from the deterministic byte plan.

Artifact: Tested KV-cache estimator

courses/ai-engineering/reference-impl/kv_cache/kv_cache_budget.py

Download reference implementation

Primary references and next links

References

1. Attention Is All You Need
Vaswani et al.. Primary transformer paper for the per-layer key/value tensor dimensions.
2. GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints
Ainslie et al.. Primary grouped-query attention paper for reduced key/value-head count.
3. Efficient Memory Management for Large Language Model Serving with PagedAttention
Kwon et al.. Primary systems paper for paged cache allocation, fragmentation, sharing, and dynamic request state.

Continue through the graph

Prefill, decode, and KV-cache interview answer →
Rehearse the compact systems explanation.
Dynamic batching trade-off →
Connect memory reservations to scheduler policy.
LLM p99 latency diagnosis →
Apply the plan to a serving incident without conflating memory and latency.

Glossary: prefill · decode · KV cache · GQA · MQA · continuous batching · paged allocation