← AI Terminology
Paged Attention
PagedAttention is a memory management algorithm for LLM inference, developed for vLLM, that manages the KV cache using a paging scheme inspired by OS virtual memory — dividing the KV cache into fixed-size pages allocated non-contiguously, eliminating fragmentation and enabling sharing across requests.
It is the core innovation that made vLLM 20–24× more throughput than naïve HuggingFace inference.
It is the core innovation that made vLLM 20–24× more throughput than naïve HuggingFace inference.
Why It Matters in AI
Traditional LLM serving pre-allocates contiguous KV cache memory per request (max sequence length × model size), wasting ~60–80% of GPU memory to fragmentation and padding. PagedAttention stores KV blocks in non-contiguous physical pages, allocated only as needed — the same approach virtual memory uses for processes. This enables far larger effective batch sizes on the same GPU, directly determining inference throughput and cost at production scale.
Key Points
| Aspect | Description |
|---|---|
| Sharing | Multiple requests sharing the same prefix (system prompt) can share physical KV pages — "prefix caching" |
| Adoption | vLLM, TGI (Text Generation Inference), SGLang, Anyscale Ray Serve — all implement PagedAttention |
| Page size | Fixed block of tokens (e.g. 16 tokens) — KV cache allocated in these page units |
| Throughput | vLLM with PagedAttention: 20–24× higher throughput than HuggingFace Transformers on same GPU |
| Block table | Per-request mapping from logical position to physical page — enables non-contiguous storage |
| Fragmentation | Naïve allocation: 60–80% memory waste; PagedAttention: ~4% waste — near-optimal utilisation |
Simple Analogy
A hotel that, instead of reserving an entire floor per guest (contiguous allocation, lots of empty rooms), assigns individual rooms as guests arrive and tracks room assignments in a ledger (block table). Every room gets used; no floor sits half-empty waiting for a guest who never comes.
Common Usage Examples
- vLLM:
vllm serve meta-llama/Meta-Llama-3-8B --gpu-memory-utilization 0.95— PagedAttention by default AsyncLLMEnginein vLLM: handles concurrent requests with shared prefix pages automatically- Prefix caching:
vllm serve ... --enable-prefix-caching— reuse KV pages for repeated system prompts - SGLang RadixAttention: extends PagedAttention with radix tree for more aggressive prefix sharing
- vLLM paper: "Efficient Memory Management for Large Language Model Serving with PagedAttention" (2023)
Summary
In short: PagedAttention manages LLM KV cache memory like OS virtual memory — storing cache in non-contiguous pages to eliminate fragmentation, enabling 20×+ throughput improvements and making efficient production LLM serving practical.