← AI Terminology
Prefill vs Decode
Prefill processes the full input prompt in parallel to build KV cache; decode generates new tokens one (or few) at a time, reading and extending that cache.
LLM inference performance analysis always splits these phases.
LLM inference performance analysis always splits these phases.
Why It Matters in AI
Prefill is compute-heavy and parallel; decode is memory-bandwidth heavy and serial. Optimisations differ (chunked prefill, prefix cache vs GQA, quantised KV). Disaggregated serving even runs them on different hardware pools.
Key Points
| Aspect | Description |
|---|---|
| Decode | One new token step; attend to all past KV |
| Metrics | TTFT ~ prefill; TPS ~ decode |
| Prefill | Prompt → parallel attention → initial KV |
| Related | KV cache, continuous batching |
| Systems | vLLM, TGI, disaggregated prefill/decode papers |
| Bottlenecks | Prefill: FLOPs; Decode: HBM bandwidth |
Simple Analogy
Reading the whole briefing book at once (prefill), then answering questions one sentence at a time while flipping back through notes (decode).
Common Usage Examples
- Profile %time prefill vs decode
- Chunked prefill for long prompts
- Disaggregated prefill workers
- KV quantisation to speed decode
Summary
In short: Prefill digests the prompt in parallel; decode emits tokens one step at a time — two phases with different bottlenecks.