← AI Terminology
Self-Attention
Self-attention is a mechanism where each token in a sequence attends to all other tokens in the same sequence — computing a weighted sum of all token representations where the weights reflect how relevant each token is to the current one.
It is the core computational operation in transformers and the key to their ability to model long-range dependencies.
It is the core computational operation in transformers and the key to their ability to model long-range dependencies.
Why It Matters in AI
RNNs model long-range dependencies by propagating information through sequential hidden states — a chain that loses signal over distance. Self-attention has direct connections between every pair of tokens regardless of distance: "the bank refused the loan because it was broke" — "it" attends directly to "bank" even if many words separate them. This O(n²) all-pairs operation enables transformers to outperform RNNs on NLP, and is why LLMs have replaced all prior sequence architectures.
Key Points
| Aspect | Description |
|---|---|
| QKV | Each token produces a Query, Key, and Value — attention weight = softmax(Q·Kᵀ/√d_k); output = weighted sum of V |
| Complexity | O(n²) in sequence length — quadratic attention makes long sequences expensive |
| Bidirectional | BERT: attend to all tokens in both directions — used for encoding/understanding tasks |
| Causal masking | Autoregressive LMs mask future tokens — token i can only attend to tokens 1…i (causal attention) |
| Efficient attn | Sparse attention (Longformer), linear attention, Mamba SSM — subquadratic alternatives |
| Flash Attention | Fused CUDA kernel: computes attention in tiles without materialising O(n²) matrix — enables long context |
Simple Analogy
A room where every person can whisper simultaneously to every other person, and the volume of each whisper is proportional to relevance: when "it" is trying to understand itself, it sends whispers and listens most to "bank" (high attention weight) and less to "the" or "and" (low weight). The resulting understanding is a weighted blend of all voices.
Common Usage Examples
nn.MultiheadAttention(d_model, num_heads)— PyTorch multi-head self-attention- Manual:
scores = Q @ K.T / sqrt(d_k); weights = softmax(scores); out = weights @ V - Causal mask:
scores.masked_fill(mask == 0, float('-inf'))— prevent attending to future tokens - Flash Attention 2:
flash_attn_func(q, k, v, causal=True)— memory-efficient long-context attention - Attention visualisation:
BertViz— heatmaps showing which tokens attend to which in BERT
Summary
In short: Self-attention allows every token to directly attend to every other token in the sequence — enabling transformers to model long-range dependencies without the sequential bottleneck of RNNs, the key operation behind all modern language models.