← AI Terminology
Attention Mechanism
Attention mechanism is a technique that lets a neural network dynamically weight how much each part of its input should influence each part of its output.
It replaced the fixed-length "bottleneck" vector used in older encoder-decoder models, allowing the network to refer back to the full input at every step.
It replaced the fixed-length "bottleneck" vector used in older encoder-decoder models, allowing the network to refer back to the full input at every step.
Why It Matters in AI
Attention is what lets a model connect a pronoun to its antecedent three paragraphs earlier, or relate a patch of sky to the word "clouds" in a caption. Without it, context was limited to whatever fit in a hidden state. The self-attention variant, used in Transformers, scales this to entire documents and is the core reason modern LLMs handle long context windows.
Key Points
| Aspect | Description |
|---|---|
| Origin | Bahdanau et al., 2014 — first applied to neural machine translation |
| Complexity | Naive self-attention is O(n²) in sequence length — Flash Attention and sparse variants address this |
| Role in AI | Central to LLMs, vision transformers, cross-modal models (image + text), protein folding |
| Key variant | Multi-head attention: runs several attention heads in parallel, each learning different relations |
| Core operation | Computes Query, Key, Value (Q, K, V) dot products; softmax over scores gives attention weights |
| Self-attention | Each token attends to every other token in the same sequence — the basis of the Transformer |
Simple Analogy
Imagine googling something: your query is matched against billions of page titles (keys), and the most relevant pages (values) are surfaced and weighted. Attention does the same thing inside the network — for every output position, it "searches" the input and blends the most relevant pieces together.
Common Usage Examples
nn.MultiheadAttentionin PyTorch — the standard building block for Transformer layers- Flash Attention (
pip install flash-attn) — memory-efficient fused CUDA kernel used by most frontier models - Cross-attention in image captioning: text decoder attends to image encoder outputs
- BERT: bidirectional self-attention lets every token see the full sentence simultaneously
- Attention maps visualised as heatmaps to interpret which words a model "focuses on"
Summary
In short: Attention is the mechanism that taught neural networks to look back at the right part of the input at the right time — and that insight powers every major AI model today.