Concept labAdvanced110 min estimateOriginal publication

The Accelerator Memory Hierarchy

Turn a slow-model complaint into a traffic ledger: which bytes move, through which tier, how often, under what reuse and capacity assumptions, and which measurement could falsify the estimate.

Authorship: InterviewsVector
Published / updated: 2026-09-26 / 2026-09-26
Review status: Artifact tests passing · primary sources recorded

Original InterviewsVector teaching. Executable artifacts are deterministic illustrative audits with focused tests and recorded primary sources; they do not claim device benchmarks, production capacity, statistical validation, hardware qualification, or universal model quality.

The decision in one pass

Accelerator performance is often governed by where data lives and how often it moves, not by the advertised arithmetic peak. Start with one phase—prefill, decode, training forward, backward, or optimizer—and inventory persistent weights, activations, KV state, temporaries, collectives, and host transfers. For every tensor, name layout, precision, lifetime, reuse, producer, consumer, and path through registers, on-chip SRAM or cache, HBM, interconnect, and host memory. Compare the arithmetic work with the bytes that must cross each constrained tier: the largest compute or transfer floor is an optimistic bound, while capacity and allocator headroom are separate admission constraints. Then measure achieved bandwidth, occupancy, stalls, kernel launches, overlap, collective time, and allocation behavior on the exact kernel, shape, software stack, topology, and device. A roofline-style estimate explains which resource deserves investigation; it is not a latency prediction until the traffic and overlap assumptions survive profiling.

Why this matters

A serving team can buy an accelerator with more FLOP/s and see little improvement because autoregressive decode repeatedly streams weights and KV state. A fused kernel can reduce HBM traffic but consume enough registers to lower occupancy. Tensor parallelism can fit a model yet replace local memory pressure with interconnect traffic. Offload can avoid an out-of-memory error while turning a millisecond path into host-transfer latency. Without a tier-by-tier ledger, these outcomes look surprising and capacity plans become vendor-spec arithmetic.

You will be able to

Map registers, on-chip SRAM or cache, HBM, interconnect, and host memory by capacity, bandwidth, latency, ownership, and programmability.
Derive phase-specific arithmetic intensity and optimistic compute or transfer floors from tensor traffic rather than parameter count alone.
Separate capacity feasibility, bandwidth limits, latency, allocator fragmentation, and scheduling effects.
Explain how tiling, fusion, recomputation, paging, parallelism, and offload exchange one resource for another.
Design a measurement plan that can falsify a traffic model on the exact deployed stack.

Prerequisite contract

Your Vector Loop for this lab

01
Model
Map one execution phase into tensor shapes, precision, layout, lifetime, reuse, kernels, tiers, device topology, and capacity ownership.
02
Derive
Derive bytes crossing each tier, arithmetic intensity, compute and transfer floors, resident-set headroom, and which overlap assumptions make the bound optimistic.
03
Build
Build a deterministic traffic ledger for an invented device and workload; calculate a full-overlap floor without presenting it as measured performance.
04
Stress
Change sequence shape, precision, fusion, reuse, KV occupancy, interconnect traffic, offload, allocator headroom, and concurrency; identify the next bottleneck.
05
Operate
Profile achieved bandwidth, stalls, occupancy, allocation, kernel mix, collective overlap, power, clocks, and tail latency on exact versions and shapes.
06
Defend
Defend what the model proves, what profiling must verify, and why more peak compute or fitting in HBM does not guarantee faster wall-clock execution.

Treat each memory tier as a different contract

Registers are private to executing threads and extremely limited; spills can turn an apparently local value into device-memory traffic. Shared memory or on-chip SRAM enables explicit reuse within a thread block but consumes capacity that can reduce concurrent blocks. Hardware caches are fast when access patterns and working sets cooperate, but their contents are not a durable application contract. HBM holds model state at high bandwidth but is far slower than on-chip storage and shared by active kernels. Interconnect and host memory introduce topology, contention, protocol, and synchronization costs that a device-local model does not capture.

Tier	Design question	Common hidden cost
registers	does each thread keep its live values local?	register pressure, spills, and reduced occupancy
on-chip SRAM/cache	is reuse tiled before eviction?	bank conflicts, synchronization, and tile-shape waste
HBM	how many bytes are read or written per useful result?	uncoalesced access, rereads, allocator headroom, and contention
interconnect	which tensors cross which links and how often?	topology bottlenecks, collectives, serialization, and imbalance
host or storage	is transfer off the request critical path?	page faults, staging copies, pinning, and bandwidth shared with control traffic

Build the traffic ledger before invoking a roofline

01Freeze the phase and shapeSeparate prefill from decode and training phases. Record batch, sequence lengths, hidden dimensions, heads, experts, precision, parallel group, and cache occupancy.
02Enumerate logical tensorsList weights, activations, KV state, optimizer state, gradients, temporaries, routing buffers, and collective payloads with lifetimes and ownership.
03Translate tensors into physical trafficAccount for reads, writes, rereads, layout conversion, padding, metadata, replication, cache hits, tiling reuse, fusion, and communication rather than counting allocation once.
04Compute per-resource floorsDivide work by achievable compute and bytes by achievable bandwidth, then state whether resources may overlap. The largest optimistic floor identifies a hypothesis, not a promise.

optimistic_time ≥ max(FLOPs / compute_rate, bytes_sram / BW_sram, bytes_hbm / BW_hbm, bytes_link / BW_link, bytes_host / BW_host)

This full-overlap form is deliberately optimistic. Dependencies, launch overhead, synchronization, occupancy, contention, topology, and ineffective overlap increase observed time; using peak rather than achieved rates makes it more optimistic still.

Arithmetic intensity is FLOPs per byte at a specified boundary. A fused attention kernel can increase HBM-level intensity by keeping tiles on chip even though the mathematical attention operation is unchanged. Always name the boundary: an intensity calculated against allocated tensor size is not the same as measured HBM traffic, and neither automatically predicts interconnect or host behavior.

Audit an optimistic tier-by-tier bound

memory_traffic_model.py

1def audit_memory_traffic(contract: MemoryTrafficContract, workload: MemoryWorkload) -> MemoryTrafficReport:
2    contract = validate_record(contract, MemoryTrafficContract)
3    workload = validate_record(workload, MemoryWorkload)
4    if workload.scope != contract.scope or workload.contract_content_id != contract.content_id:
5        raise ValueError("workload belongs to another memory traffic contract")
6    # Aggregate declared bytes per tier, calculate optimistic transfer floors,
7    # then check HBM residency headroom separately from throughput.

Expected output

example=illustrative_only
decision=WITHIN_DECLARED_BOUND
limiting_resource=hbm
optimistic_token_floor_us=8000
optimistic_tokens_per_second_ceiling=125
claim=LOCAL_OPTIMISTIC_TRAFFIC_BOUND_NOT_DEVICE_BENCHMARK

Verify: python3 -m unittest discover courses/ai-engineering/reference-impl/accelerator_memory

The invented fixture declares four canonical tiers, a six-millisecond compute floor, four per-token traffic legs, 48 GB of resident HBM state, and 20 percent capacity headroom. Sixteen billion declared HBM bytes at an invented 2 TB/s produce the limiting eight-millisecond optimistic floor and an arithmetic ceiling of 125 tokens/s. The implementation copies frozen records, aggregates multiple legs per tier, binds scope and content digests, and rejects duplicate identities, missing or reordered tiers, malformed types, booleans as integers, constructor bypass, and stale evidence.

Nothing runs on a device. The values do not describe a commercial accelerator or real model, and the full-overlap ceiling ignores kernel efficiency, contention, allocation, synchronization, clocks, thermals, and topology. Its purpose is to make traffic and headroom assumptions reviewable before profiling.

Locate the active memory bottleneck

Inspect an invented compute floor, tier bandwidth, per-token traffic, and resident set. Predict the limiting resource before revealing the arithmetic bound. No accelerator is queried.

Locate the limiting level before tuning

Read the movement path from on-chip reuse through HBM, device links, and capacity. Predict the first intervention from the measured bottleneck rather than from peak hardware specifications.

Memory-path trace

Tensor cores wait on HBMPeak arithmetic is available, but weights stream repeatedly from high-bandwidth memory.A shard boundary floods the interconnectCompute and local HBM have headroom while collective transfers dominate each token step.KV growth exhausts device capacityLong contexts fit at low concurrency, then allocations fail before throughput limits are reached.The kernel reaches the compute roofData is reused effectively and neither HBM nor links approach their operating limits.

Devices, kernels, rates, and thresholds are synthetic. This lab does not benchmark or configure real hardware.

Hierarchy evidence
Signal	Operating constraint	Observed evidence
Tensor-core utilization	Investigate below 55%	31%
HBM bandwidth	Sustained use below 90%	94% of measured peak
Arithmetic intensity	Roofline knee at 96 FLOP/byte	24 FLOP/byte
L2 hit rate	Target at least 70%	38%

Illustrative measurements only. The recommended action stays hidden until you check a prediction.

Which intervention addresses the active constraint?

Increase on-chip reuse before adding computeReduce cross-device transfers and restore localityReduce the resident memory footprintOptimize arithmetic throughput first

Choose an action before checking the operating contract.

Move the bottleneck deliberately

Technique	Resource reduced	Resource or risk increased
kernel fusion	HBM round trips and launches	register/SRAM pressure, compile complexity, and larger failure surface
tiling	lower-tier rereads	boundary waste, synchronization, and shape sensitivity
recomputation	activation storage and reads	compute, energy, and possibly latency
quantization	weight/KV bytes and bandwidth	conversion, kernel coverage, calibration, and numerical error
tensor parallelism	per-device weights and compute	collective traffic, link contention, and stragglers
host offload	device capacity	host/link traffic, page management, and tail latency

After an optimization, rebuild the entire ledger. Removing HBM traffic may expose compute, on-chip capacity, or interconnect as the next limit. The goal is not to prove the old bottleneck disappeared in isolation; it is to improve the product metric under the complete workload and operating envelope.

Profile the deployed path and preserve the envelope

Benchmark warm and cold paths across representative prompt, output, batch, cache, and concurrency distributions. Capture exact device, topology, driver, runtime, compiler, kernel, model, quantization, scheduler, power, and clock versions. Reconcile requested tensor sizes with allocator reservations and fragmentation. Compare estimated bytes with profiler counters, and compare device time with wall-clock latency so queueing, CPU work, data movement, and synchronization stay visible.

Alert on HBM reservation and fragmentation, achieved bandwidth, occupancy, spill signals, collective time, host transfer, throttling, and kernel-version drift.
Keep capacity headroom for allocator behavior, temporary workspaces, KV growth, uneven shards, recovery, and traffic bursts rather than planning at nominal capacity.
Treat profiler access and captured tensors as sensitive operational data with bounded retention and tenant-aware controls.
Re-run the matrix after model, kernel, precision, context, scheduler, driver, topology, or device changes; old measurements do not transfer automatically.

Operate at three altitudes

Production lens

— Track achieved rather than advertised compute and bandwidth alongside spills, occupancy, HBM reservation, fragmentation, collectives, host transfer, clocks, power, and end-to-end latency by exact workload shape.
— Capacity-plan weights, KV state, activations, workspaces, allocator behavior, uneven partitions, failover, and burst headroom together; an average resident set is not a safe admission limit.
— Keep a reproducible profile matrix and regression thresholds across device, topology, driver, runtime, compiler, kernel, model, precision, scheduler, and traffic versions.

Staff lens

— Make product, model, compiler, kernel, serving, and infrastructure owners share one traffic and capacity ledger so local optimizations cannot externalize cost invisibly.
— Use analytical bounds to choose measurements and architecture experiments, then require measured end-to-end evidence before procurement or SLO commitments.

Interview defense

A new accelerator advertises twice the compute, but decode throughput barely changes. How do you investigate?

I would freeze the decode shape and versioned stack, then build a per-token ledger for weights, KV reads and writes, activations, collectives, and host traffic. I would compare compute and tier-specific transfer floors using achieved—not only peak—rates, and check HBM capacity headroom separately. Decode may be streaming weights or growing KV state, so more FLOP/s will not help if HBM or interconnect is limiting. I would profile achieved bandwidth, cache behavior, occupancy, register spills, kernel mix, collectives, clocks, allocator state, and wall-clock queueing. Then I would test a bottleneck-specific change such as fusion, quantization, batching, or topology and rebuild the ledger because the limit may move.

Expect the interviewer to press on

— Why is allocated tensor size not the same as memory traffic?
— How can fusion reduce traffic yet lower performance?
— Which assumptions make a roofline estimate optimistic?

Misconceptions to remove

“If the model fits in HBM, memory is no longer a performance concern.”

Capacity only shows that a resident set may fit. Repeated weight, KV, activation, and temporary traffic can still dominate time, and allocator headroom or fragmentation can still break admission.

“Peak FLOP/s divided by model FLOPs predicts latency.”

That ignores tier traffic, reuse, kernel efficiency, occupancy, synchronization, communication, shape, queueing, and wall-clock work outside the accelerator.

“A lower-precision tensor always moves proportionally faster.”

The deployed kernel may convert formats, lack coverage, become compute-bound, use different tiling, or expose another tier. Measure the exact end-to-end path and quality envelope.

Check your model

1. What boundary must accompany an arithmetic-intensity value?

The memory tier or interface whose bytes are counted—such as HBM or interconnect—plus the execution phase and reuse assumptions.

2. Why is max(compute time, tier transfer times) optimistic?

It assumes ideal rates and full overlap without dependency, launch, synchronization, occupancy, contention, topology, allocation, or scheduling penalties.

3. What changes after an optimization removes the current bottleneck?

Another resource becomes limiting, so the complete traffic and capacity ledger and end-to-end measurement must be rebuilt rather than extrapolating the old model.

Prove the mechanism

Extend the artifact with separate prefill and decode workloads plus an explicit non-overlap mode. Preserve integer arithmetic, scope binding, headroom, and stable evidence while showing how the limiting tier changes.

Add a production constraint

Produce a falsifiable traffic and capacity model for a sharded model across two hosts. Include weights, KV, activations, collectives, host staging, failure headroom, topology, overlap assumptions, profiler counters, and a benchmark matrix for prompt and output distributions.

Artifact: Accelerator memory traffic model

courses/ai-engineering/reference-impl/accelerator_memory/memory_traffic_model.py

Download reference implementation

Primary references and next links

References

1. CUDA C++ Programming Guide
NVIDIA. Official documentation on CUDA execution, device memory spaces, hierarchy, synchronization, and programming behavior.
2. Roofline: An Insightful Visual Performance Model for Multicore Architectures
Williams, Waterman, and Patterson. Primary research connecting operational intensity with compute and memory-bandwidth ceilings.
3. FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness
Dao et al.. Primary research analyzing exact attention through IO complexity and tiling across GPU memory hierarchy.

Continue through the graph

The KV Cache Capacity Plan →
Translate per-token KV state into the resident set and HBM traffic of serving workloads.
Precision and Quantization as Error Budgets →
Trade tensor bytes and kernel behavior against a measured numerical and product-quality budget.

Glossary: memory hierarchy · arithmetic intensity · roofline model · HBM · on-chip SRAM · register spill · occupancy · memory coalescing · interconnect · allocator fragmentation