← AI Terminology
Cross-Attention
Cross-attention is attention where queries come from one sequence and keys/values from another (e.g. decoder reading encoder states, or text conditioning an image model).
It is the fusion bridge in encoder–decoder models, multimodal systems, and conditioned generators.
It is the fusion bridge in encoder–decoder models, multimodal systems, and conditioned generators.
Why It Matters in AI
Self-attention mixes one stream; cross-attention lets one modality or stage read another — translation decoders reading source text, captioners reading vision tokens, diffusion models reading prompts.
Key Points
| Aspect | Description |
|---|---|
| Caches | Encoder K/V can be computed once per input |
| Related | Q-Former, Perceiver resamplers as bottleneck cross-attn |
| Diffusion | U-Net/DiT cross-attn to text embeddings |
| Multimodal | LLM queries over vision encoder tokens |
| Classic use | Transformer decoder attending to encoder outputs |
| Vs self-attention | Q from A, K/V from B — different sequences |
Simple Analogy
A student (queries) looking up facts in a reference book (keys/values) rather than only rereading their own notes.
Common Usage Examples
- T5/BART decoder cross-attention layers
- Stable Diffusion: cross-attn to CLIP text embeddings
- Whisper decoder attending to audio encoder states
MultiheadAttentionwith separate Q and KV sources
Summary
In short: Cross-attention lets one sequence read another — the fusion mechanism behind translation, captioning, and text-conditioned generation.