← AI Terminology
RoPE - Rotary Position Embedding
RoPE (Rotary Position Embedding) encodes positions by rotating query and key vectors in 2D planes as a function of position — relative position appears naturally in attention scores.
Introduced in RoFormer and adopted by almost every modern decoder LLM (LLaMA, Mistral, GPT-NeoX-style stacks).
Introduced in RoFormer and adopted by almost every modern decoder LLM (LLaMA, Mistral, GPT-NeoX-style stacks).
Why It Matters in AI
Absolute learned position embeddings struggle with long context; RoPE injects relative geometry into attention with strong extrapolation (plus scaling tricks). It is the positional encoding you must know for long-context LLMs.
Key Points
| Aspect | Description |
|---|---|
| Origin | Su et al., RoFormer (2021) |
| Used in | LLaMA, Mistral, Qwen, most open decoder-only models |
| Mechanism | Pair dims of q/k; apply rotation R_θ,m at position m |
| Alternatives | ALiBi, T5 relative bias, classic sin/cos encodings |
| Long context | NTK-aware, YaRN, PI rescale frequencies for longer sequences |
| Relative property | q_mᵀ k_n depends on (m−n) after rotations |
Simple Analogy
Instead of stamping each seat with a fixed number, you rotate each person's compass by how far they sit — pairwise angles encode relative distance.
Common Usage Examples
- LLaMA
apply_rotary_pos_emb - Fused RoPE in FlashAttention / vLLM
- YaRN / NTK-aware scaling for long context
- Needle-in-a-haystack tests after RoPE scaling
Summary
In short: RoPE encodes position by rotating Q/K vectors so attention naturally sees relative distances — the positional backbone of modern LLMs.