← AI Terminology
Sliding Window Attention
Sliding window attention restricts each token to attend only to a fixed local neighbourhood of tokens, often stacked across layers so the effective receptive field grows with depth.
Used in Longformer-style models and efficient decoders such as early Mistral configurations.
Used in Longformer-style models and efficient decoders such as early Mistral configurations.
Why It Matters in AI
Full attention is O(n²). Local windows make long sequences tractable while deep stacks expand range. Production long-context models often mix windowed layers with occasional global or full attention.
Key Points
| Aspect | Description |
|---|---|
| Serving | Bounded KV per token helps memory at long n |
| Examples | Longformer, Mistral windowed attention, some Gemma variants |
| Tradeoff | May miss ultra-long deps without global tokens |
| With RoPE | Window + rotary positions is common |
| Complexity | O(n · w) for window size w vs O(n²) |
| Effective range | Roughly layers × window after stacking |
Simple Analogy
Each person in a line may only talk to w neighbours; after many rows of such conversations, information still propagates far down the line.
Common Usage Examples
- Mistral early configs: window size 4096
- Longformer: windowed + global task tokens
- FlashAttention local/window variants
- Set attention window in long-document models
Summary
In short: Sliding window attention limits each token to a local neighbourhood — linear-cost attention that still spans long documents when stacked deep.