← AI Terminology
KV-Cache Quantization
KV-cache quantization stores attention key/value caches in lower precision (INT8/FP8/INT4) during generation to cut memory and bandwidth.
Critical for long-context and high-concurrency serving.
Critical for long-context and high-concurrency serving.
Why It Matters in AI
Decode is bandwidth-bound by KV reads. Quantising caches often yields large free throughput wins with small quality loss when done carefully — a hot inference-engineering lever.
Key Points
| Aspect | Description |
|---|---|
| Why | KV dominates memory at long n and large batch |
| Pair | GQA/MLA already shrink KV |
| Risk | Quality loss; outliers need special handling |
| Engines | vLLM, TensorRT-LLM, research kernels |
| Formats | INT8/FP8 common; research on 4-bit KV |
| Related | Paged attention, prefill vs decode |
Simple Analogy
Keeping meeting notes in shorthand instead of full transcripts — much thinner folder, almost the same ability to recall decisions.
Common Usage Examples
- Enable KV quant flags in serving engines
- Measure TPS and MMLU-style regressions
- Long-context stress tests
- Combine with GQA
Summary
In short: KV-cache quantization stores attention caches in low precision — freeing memory bandwidth for faster long-context decode.