← AI Terminology
Sparse Attention
Sparse attention computes only a subset of token-pair interactions (local, strided, random, or learned patterns) instead of the full n×n matrix.
It is a family of methods for scaling transformers to longer sequences.
It is a family of methods for scaling transformers to longer sequences.
Why It Matters in AI
Quadratic attention limits context length. Sparse patterns trade density for length — Longformer, BigBird, Sparse Transformer, and long-document systems. Pattern choice matters for books, codebases, and genomics.
Key Points
| Aspect | Description |
|---|---|
| Today | Often replaced by GQA+window+SSM hybrids for LLMs |
| Quality | Task-dependent; full attention still best when affordable |
| Hardware | Irregular sparsity can be GPU-unfriendly vs structured windows |
| Patterns | Local window, dilated/strided, global tokens, random (BigBird) |
| Complexity | Often O(n√n) or O(n·w) |
| Classic papers | Sparse Transformer, Longformer, BigBird, Reformer |
Simple Analogy
Instead of every citizen calling every other citizen, you only call neighbours, a few hubs, and some random long-distance contacts — still connected, far fewer calls.
Common Usage Examples
- BigBird/Longformer in Hugging Face
- Block-sparse attention research stacks
- Reformer LSH attention
- Compare long-bench scores vs full attention
Summary
In short: Sparse attention prunes which token pairs interact — the classic path to longer contexts when full quadratic attention is too expensive.