← AI Terminology
GQA - Grouped-Query Attention
GQA (Grouped-Query Attention) shares key/value heads across groups of query heads — a middle ground between multi-head attention (MHA) and multi-query attention (MQA).
It cuts KV-cache size and bandwidth during decode while preserving most quality of full MHA.
It cuts KV-cache size and bandwidth during decode while preserving most quality of full MHA.
Why It Matters in AI
Autoregressive decode is memory-bandwidth bound by the KV cache. GQA shrinks that cache with far less quality loss than pure MQA, enabling longer context and higher throughput. LLaMA 2/3, Mistral, and many production models ship with GQA.
Key Points
| Aspect | Description |
|---|---|
| GQA | H_kv = H_q / g — e.g. 32 query heads, 8 KV heads |
| MHA | H_q = H_kv — full multi-head (largest KV cache) |
| MQA | H_kv = 1 — maximum savings, more quality risk |
| Origin | Ainslie et al., GQA (2023) |
| Benefit | Faster decode, smaller memory, near-MHA quality |
| Used in | LLaMA 2/3, Mistral 7B, Gemma, many open models |
Simple Analogy
Instead of every reader keeping a full personal notebook (MHA), reading groups share one notebook (KV) while each still asks their own questions (Q).
Common Usage Examples
- LLaMA-2-70B: 64 query heads, 8 KV heads
- HF config:
num_key_value_heads - KV cache ∝ num_key_value_heads × seq_len × layers
- Compare decode TPS: MHA vs GQA vs MQA
Summary
In short: GQA groups query heads to share KV heads — how modern LLMs shrink the KV cache without giving up multi-head quality.