vLLM and PagedAttention
Efficient Memory Management for Large Language Model Serving with PagedAttention
Woosuk Kwon, Zhuohan Li, Siyuan Zhuang, Ying Sheng, Lianmin Zheng, Cody Hao Yu, Joseph E. Gonzalez, Hao Zhang, Ion Stoica
The paper in one sentence
PagedAttention makes LLM serving more efficient by allocating KV-cache memory in blocks instead of reserving large contiguous regions per request.
What the paper does
PagedAttention applies ideas from virtual memory to the key-value cache used during autoregressive generation. vLLM stores cache data in non-contiguous blocks, reducing fragmentation and enabling flexible sharing across requests and decoding strategies.
Why it matters to engineers
KV-cache capacity is often the limiting resource in high-throughput LLM serving. Better memory utilization lets a serving system batch more active sequences while preserving latency targets.
Key claims from the paper
- Block-based KV-cache allocation can reduce memory waste to near zero.
- Paged memory enables flexible cache sharing within and across requests.
- The vLLM system reports two-to-four-times higher throughput than evaluated serving baselines at similar latency.
Architecture, step by step
- 1
Logical blocks
Each sequence views its KV cache as an ordered set of logical blocks.
- 2
Physical blocks
A memory manager maps logical blocks to available non-contiguous GPU blocks.
- 3
Attention kernel
PagedAttention follows the block table while reading keys and values for the active sequence.
- 4
Continuous batching
Freed blocks can be reassigned quickly as requests finish and new decoding steps enter the batch.
Production takeaways
- Capacity planning should model KV-cache bytes per token, model geometry, precision, and concurrent sequence length.
- Continuous batching improves utilization but needs admission control to protect tail latency.
- Prefix sharing and copy-on-write can reduce duplicated cache data for parallel sampling or shared prompts.
- Benchmark realistic prompt/output distributions; uniform synthetic lengths can hide fragmentation and queuing effects.
Limitations and caveats
- Reported gains depend on model, hardware, workload distribution, baseline, and latency objective.
- Memory efficiency does not eliminate compute bottlenecks or overload-driven queuing.
- A production serving stack still needs scheduling, failure isolation, observability, and safe rollout controls.
Interview questions to test understanding
- Why does KV-cache fragmentation reduce serving throughput?
- How is PagedAttention analogous to virtual memory?
- What trade-off does continuous batching create for tail latency?
- How would you estimate the maximum number of concurrent sequences on one GPU?