← AI Terminology
Positional Encoding
Positional encoding is a technique that injects information about token position into transformer input embeddings — allowing the model to distinguish "dog bites man" from "man bites dog" — since attention is otherwise permutation-invariant and cannot detect token order.
It is a required component of every transformer architecture.
It is a required component of every transformer architecture.
Why It Matters in AI
Unlike RNNs, transformers process all tokens simultaneously in parallel. This parallelism is efficient but destroys positional information: without explicit position signals, "the cat sat on the mat" and "mat the on sat cat the" look identical to attention. Positional encoding solves this by adding position-dependent vectors to token embeddings. The choice of encoding scheme significantly affects long-context performance — RoPE is why Llama 3 can handle 128K context lengths.
Key Points
| Aspect | Description |
|---|---|
| RoPE | Rotary Position Embedding: encode position as rotation in embedding space — used in Llama, GPT-NeoX |
| ALiBi | Add bias to attention scores based on distance — enables extrapolation beyond training length |
| Relative | Encode distance between tokens, not absolute position — T5, DeBERTa |
| Learned absolute | BERT, GPT-2: learnable position embedding table — one learned vector per position |
| Context extension | RoPE + YaRN/LongRoPE: interpolate or extrapolate position embeddings to support longer contexts |
| Absolute (sinusoidal) | Original Transformer: fixed sin/cos patterns of different frequencies per position |
Simple Analogy
Seat numbers in a cinema: the film (content) is the same, but knowing you're in row 5, seat 10 vs. row 15, seat 3 changes how you experience the surrounding context. Positional encoding stamps each token's "seat number" onto its representation so the model knows who's sitting where.
Common Usage Examples
- Sinusoidal:
PE(pos, 2i) = sin(pos/10000^(2i/d_model))— original Transformer formula nn.Embedding(max_seq_len, d_model)— learned absolute position embeddings (BERT style)- Llama 3 RoPE:
apply_rotary_emb(q, k, freqs_cis)— rotate Q and K by position-dependent complex numbers - ALiBi:
attn_bias = -m × |i-j|— attention bias proportional to distance, slope m per head - Context extension:
rope_scaling = {"type": "yarn", "factor": 4.0}— extend 32K→128K context
Summary
In short: Positional encoding injects token order information into permutation-invariant transformers — a required component of every LLM, with RoPE being the modern standard that enables efficient long-context training and extension.