← AI Terminology
Prefix Caching
Prefix caching stores the KV cache (or equivalent state) for a token prefix so later requests sharing that prefix skip re-prefill of the shared part.
It is the systems mechanism behind many prompt-caching products.
It is the systems mechanism behind many prompt-caching products.
Why It Matters in AI
Prefill is expensive for long contexts. Prefix caching makes multi-turn and multi-tenant serving efficient when prompts share heads (same system prompt, same document). Inference engineers tune block sizes and eviction for hit rate.
Key Points
| Aspect | Description |
|---|---|
| Engines | vLLM, TensorRT-LLM, SGLang, major APIs |
| Related | Prompt caching, paged attention |
| Eviction | LRU of KV blocks under VRAM pressure |
| Hit rate | Depends on stable leading tokens |
| Mechanism | Hash/store KV blocks for token prefixes |
| Multi-turn | Prior turns become the growing prefix |
Simple Analogy
Leaving a bookmark and partial notes in a shared textbook chapter so the next student starting the same chapter skips rereading from page one.
Common Usage Examples
- vLLM automatic prefix caching metrics
- Multi-turn chat reuses prior KV
- Shared system prompt across users
- Monitor cache hit rate in serving dashboards
Summary
In short: Prefix caching keeps KV state for shared prompt heads so the server does not re-prefill the same tokens every time.