← AI Terminology
Multi-Head Attention
Multi-head attention is the attention mechanism used in transformers that runs multiple parallel attention operations ("heads"), each learning to attend to different aspects of the input, and concatenates their outputs — giving the model richer representational capacity than a single attention operation.
It is the core computational primitive of every transformer architecture.
It is the core computational primitive of every transformer architecture.
Why It Matters in AI
A single attention head learns one way to relate tokens — one type of dependency (e.g. subject-verb agreement). Multi-head attention runs H independent heads simultaneously, each capturing different relationships (syntax, coreference, positional proximity, semantic similarity), then combines them. This multiplicative representational capacity is why transformers outperform RNNs on long-range dependencies. Without multi-head attention, LLMs as we know them would not exist.
Key Points
| Aspect | Description |
|---|---|
| Heads H | Typical values: 8, 12, 16, 32 — each head operates on a d_head = d_model / H dimensional subspace |
| Q, K, V | Each head has its own learned projections for queries, keys, and values |
| GQA / MQA | Grouped-Query Attention: multiple query heads share K/V heads — reduces KV cache (Llama 3, Gemma) |
| Concatenation | Head outputs concatenated and projected back to d_model — single output per position |
| Flash Attention | Fused kernel that computes multi-head attention without materialising the full attention matrix |
| Scaled dot-product | Attention(Q,K,V) = softmax(QKᵀ/√d_k) × V — score each query against all keys, weighted-sum values |
Simple Analogy
A panel of experts reviewing a document: each expert (head) focuses on a different aspect — one reads for factual claims, one for logical structure, one for tone — and they compare notes at the end. One expert reading for everything would miss the nuances that each specialist catches.
Common Usage Examples
nn.MultiheadAttention(d_model=512, num_heads=8)— PyTorch built-in multi-head attention- Transformer block:
x = x + self.attn(self.norm1(x))— residual connection around MHA - Visualising attention:
BertViz— plots attention weights per head for BERT models - Flash Attention 2:
flash_attn_func(q, k, v, causal=True)— memory-efficient MHA kernel - Llama 3: 32 query heads, 8 KV heads (GQA) — 4× KV cache reduction vs. standard MHA
Summary
In short: Multi-head attention runs parallel attention operations that each capture different token relationships, then combines them — the fundamental computational building block of every transformer, enabling LLMs to model complex long-range dependencies.