← AI Terminology
Attention Sink
Attention sink refers to the finding that early tokens (especially the first) attract disproportionate attention mass, acting as a stabilising 'sink' in long or streaming contexts.
StreamingLLM shows that keeping sink tokens plus a recent window preserves quality when dropping middle KV states.
StreamingLLM shows that keeping sink tokens plus a recent window preserves quality when dropping middle KV states.
Why It Matters in AI
Naively evicting old KV entries breaks models because attention expects certain initial tokens to remain. Recognising sinks enables infinite-stream decoding with finite memory — critical for long sessions.
Key Points
| Aspect | Description |
|---|---|
| Why | Softmax must sum to 1; sinks absorb leftover mass |
| Paper | StreamingLLM (Xiao et al., 2023) |
| Related | Window attention, rolling buffers, infinite-context serving |
| Practice | Do not casually drop BOS/position-0 from cache |
| Observation | Softmax dumps mass on initial tokens even if weak semantically |
| StreamingLLM | Keep a few sink tokens + sliding recent KV window |
Simple Analogy
In a meeting, people keep glancing at the chairperson even when discussion moved on — that fixed gaze stabilises the room’s attention budget.
Common Usage Examples
- StreamingLLM policy: sinks + last L tokens
- Long-chat serving without OOM
- Attention maps: bright column at token 0
- Combine with sliding window in production
Summary
In short: Attention sinks are early tokens that absorb excess attention mass — keep them in cache if you want streaming LLMs to stay stable when forgetting the middle.