← AI Terminology
Encoder-Decoder
An encoder-decoder architecture is a neural network design where an encoder compresses input into a latent representation and a decoder generates output from that representation — enabling sequence-to-sequence transformations like translation, summarisation, and image generation.
The original Transformer (2017) used an encoder-decoder for machine translation.
The original Transformer (2017) used an encoder-decoder for machine translation.
Why It Matters in AI
The encoder-decoder split is the foundational architecture for tasks where the input and output have different structures or lengths. Machine translation (English → German), summarisation (article → summary), image captioning (image → text), and audio transcription (speech → text) all use encoder-decoder models. Variants — encoder-only (BERT) and decoder-only (GPT) — dominate different sub-fields.
Key Points
| Aspect | Description |
|---|---|
| Decoder | Autoregressively generates output tokens, conditioned on encoder representation |
| Encoder | Processes input, produces contextual embeddings — no output tokens generated here |
| Decoder-only | GPT — generates text; no separate encoder; uses causal attention |
| Encoder-only | BERT — bidirectional, no decoder; excels at classification, NER, QA |
| Enc-dec models | T5, BART, Whisper (ASR), MarianMT — best for tasks with distinct input/output modalities |
| Cross-attention | Decoder attends to encoder output — the mechanism linking encoded input to decoded output |
Simple Analogy
A professional interpreter: the encoder is the process of fully understanding the speaker's message (no speaking yet — just comprehension); the decoder is the act of reformulating and speaking that message in another language, word by word, checking back against the understood meaning at each step (cross-attention).
Common Usage Examples
T5ForConditionalGeneration— encode prompt, decode generated text:outputs = model.generate(input_ids)- Whisper: audio encoder (Transformer) → text decoder (Transformer) — state-of-the-art ASR
- BART: encoder-decoder pre-trained with denoising, fine-tuned for summarisation and translation
- Stable Diffusion: CLIP text encoder → U-Net diffusion decoder (conditioned on text encoding)
MarianMTModelin HuggingFace: encoder-decoder for 1,000+ language-pair translations
Summary
In short: Encoder-decoder architectures separate understanding (encoder) from generation (decoder) — the foundational design for any task where input and output are different in type or length.