← AI Terminology
Linear Attention
Linear attention reformulates attention so cost scales roughly O(n) in sequence length, typically via kernel feature maps that avoid materialising the full attention matrix.
Examples include the Linear Transformer, Performer, and related kernelised methods.
Examples include the Linear Transformer, Performer, and related kernelised methods.
Why It Matters in AI
If attention is linearised, very long contexts become more plausible. Quality has historically lagged softmax attention, but the idea influences SSMs and efficient sequence layers.
Key Points
| Aspect | Description |
|---|---|
| Cons | Often weaker than softmax on language modelling |
| Idea | φ(Q)(φ(K)ᵀV) associativity avoids the n×n matrix |
| Pros | Long sequence scaling; recurrent-style inference possible |
| Status | Research and niches; softmax + systems tricks still dominate LLMs |
| Examples | Performer (FAVOR+), Linear Transformer, CosFormer |
| Relation | Connects to SSMs/RNNs as efficient sequence mixers |
Simple Analogy
Instead of comparing every pair of people pairwise, you project everyone into summary features and combine summaries — cost grows with people, not pairs.
Common Usage Examples
- Performer attention in long-sequence experiments
fast_transformerslinear attention- Hybrid: local softmax + global linear
- Length-scaling ablations vs softmax
Summary
In short: Linear attention approximates or kernelises attention to run in roughly O(n) time — a key efficient-sequence idea even when softmax still wins quality.