← AI Terminology
MLA - Multi-Head Latent Attention
MLA (Multi-Head Latent Attention) is DeepSeek’s attention variant that compresses keys and values into a low-rank latent cache, then up-projects for attention — cutting KV size while retaining multi-head expressivity.
It is a signature component of DeepSeek-V2/V3-style architectures.
It is a signature component of DeepSeek-V2/V3-style architectures.
Why It Matters in AI
Long-context and MoE models hit memory walls on KV storage. MLA compresses latents rather than only reducing head count (GQA), enabling cheaper long context at large scale — a key idea in the 2024–2025 open-model wave.
Key Points
| Aspect | Description |
|---|---|
| Origin | DeepSeek-V2 technical report |
| Status | Architecture-specific; less ubiquitous than GQA so far |
| Benefit | Much smaller cache with strong quality at DeepSeek scale |
| Pairing | Often combined with MoE FFNs in DeepSeek models |
| Related | Low-rank KV, GQA, multi-head sharing |
| Core idea | Store compressed latent c_KV instead of full multi-head K/V |
Simple Analogy
Instead of filing every full document, you keep a dense summary card per page and reconstruct detail only when answering a question.
Common Usage Examples
- DeepSeek-V2/V3 model cards describe MLA
- Compare KV bytes/token: MHA vs GQA vs MLA
- Long-context cost discussions in DeepSeek reports
- Open reproductions of latent KV layouts
Summary
In short: MLA compresses KV into latent vectors so multi-head attention runs with a much smaller cache — DeepSeek’s answer to long-context memory pressure.