← AI Terminology
KV Cache
KV cache (Key-Value cache) is an inference optimisation for transformer models that stores the key and value tensors computed for all previous tokens — so that when generating each new token, the model reuses cached computations rather than reprocessing the entire context from scratch.
Without it, autoregressive generation would recompute the full attention matrix for every new token.
Without it, autoregressive generation would recompute the full attention matrix for every new token.
Why It Matters in AI
KV cache is what makes LLM inference practical: without it, generating a 1,000-token response would require 1,000 separate full-context forward passes. With it, each new token only needs one new K/V pair per layer, reducing generation compute from O(n²) to O(n) per step. It is the single most important optimisation in LLM serving — but also the largest memory cost, driving innovations like PagedAttention (vLLM) and multi-query attention.
Key Points
| Aspect | Description |
|---|---|
| Memory cost | KV cache grows linearly with sequence length × batch size × layers × heads — the bottleneck in LLM serving |
| Quantisation | INT8/FP8 KV cache — halves or quarters memory with minimal quality loss |
| PagedAttention | vLLM's innovation: manages KV cache like OS virtual memory — allows larger effective batch sizes |
| Prefix caching | Cache the KV of a repeated system prompt — reuse across requests (Claude, OpenAI support this) |
| What is cached | Key and Value matrices from the attention mechanism for all previous tokens in the sequence |
| Multi-Query Attn | MQA / GQA: share K/V heads across query heads — reduces KV cache size 4–8× (used in Llama 3) |
Simple Analogy
A translator working through a document: instead of re-reading every prior sentence each time they write a new one, they keep a notepad of key points and context already processed. Each new sentence only requires reading the latest line and checking the notepad — the KV cache is that notepad.
Common Usage Examples
- vLLM PagedAttention: dynamically allocates KV cache pages, eliminating fragmentation and doubling throughput
use_cache=Truein HuggingFacemodel.generate()— enabled by default for autoregressive generation- Llama 3 uses Grouped-Query Attention (GQA) to reduce KV cache by sharing K/V across 8 query groups
- Prefix caching:
anthropic.beta.prompt_caching— caches system prompt KV across API calls transformers.Cacheclass:DynamicCache,StaticCache— different KV cache implementations
Summary
In short: The KV cache stores all previously computed attention keys and values so each new token only needs one incremental computation — the essential optimisation that makes autoregressive LLM generation fast and economical.