← AI Terminology
Transformer
Transformer is a neural network architecture that processes sequences by learning which parts of the input to pay attention to, rather than reading tokens one at a time like earlier recurrent models.
It was introduced in the 2017 paper *"Attention Is All You Need"* by Google researchers.
It was introduced in the 2017 paper *"Attention Is All You Need"* by Google researchers.
Why It Matters in AI
The Transformer unlocked training on far larger datasets with far more parallelism than RNNs or LSTMs allowed, which directly enabled the large language model era (GPT, BERT, Gemini, Claude). Its attention mechanism generalises well beyond text — images, audio, protein sequences, and video all now use Transformer variants. Nearly every frontier AI model today is built on this architecture.
Key Points
| Aspect | Description |
|---|---|
| Origin | Google Brain, 2017 ("Attention Is All You Need" — Vaswani et al.) |
| Frameworks | PyTorch nn.Transformer, HuggingFace transformers, JAX/Flax, TensorFlow Keras |
| Role in AI | Foundation of every major LLM, vision transformer (ViT), and multimodal model |
| Competitors | State space models (Mamba/SSM) are the main architectural alternative for long-context efficiency |
| Key variants | Encoder-only (BERT), decoder-only (GPT series), encoder-decoder (T5, original translation models) |
| Core mechanic | Self-attention: each token learns how much to "attend" to every other token in the context |
Simple Analogy
Imagine reading a long legal contract: instead of reading every word in strict order, a skilled lawyer scans the whole document and highlights which clauses relate to each other. Self-attention is that highlighting — every word simultaneously checks its relevance to every other word, then combines the results.
Common Usage Examples
from transformers import AutoModelForCausalLM— load any open-weight LLM in one line- Fine-tuning BERT for sentiment classification
- Vision Transformer (ViT): image split into 16×16 patches fed as tokens
- Whisper (OpenAI) uses a Transformer encoder-decoder for speech-to-text
- AlphaFold 2 applies attention to protein residue sequences for structure prediction
Summary
In short: The Transformer is the single architectural invention that made modern AI — from ChatGPT to image generators — possible.