← AI Terminology
BERT - Bidirectional Encoder Representations from Transformers
BERT stands for Bidirectional Encoder Representations from Transformers: a Transformer encoder pre-trained by Google (2018) using masked language modeling — predicting randomly hidden words using context from both directions.
It established the "pre-train then fine-tune" paradigm that dominates NLP today.
It established the "pre-train then fine-tune" paradigm that dominates NLP today.
Why It Matters in AI
Before BERT, NLP models were trained from scratch on small task-specific datasets. BERT's key insight was pre-training a large Transformer on massive unlabelled text and then fine-tuning with a small labelled dataset — achieving state-of-the-art on 11 NLP tasks at once. It broke the dam: GPT, RoBERTa, ALBERT, DeBERTa, and eventually GPT-4 all follow this pre-train/fine-tune blueprint.
Key Points
| Aspect | Description |
|---|---|
| Variants | RoBERTa (more data, no NSP), ALBERT (param sharing), DistilBERT (knowledge distilled, 40% smaller) |
| Limitation | Encoder-only — excels at understanding tasks (classification, NER), not generation |
| Fine-tuning | Add a task head; train on labelled data — classification, NER, QA, all work well |
| Architecture | Transformer encoder only (bidirectional) — reads the full sequence in both directions at once |
| Displaced by | GPT-class decoder models for generation; still dominant for embedding/retrieval tasks |
| Pre-training | Masked LM (predict [MASK] tokens) + Next Sentence Prediction on 3.3B words of text |
Simple Analogy
BERT reads a sentence like a human re-reading — it sees the full context in both directions before predicting any word. Earlier models read left-to-right only, like a typewriter. BERT is the reader who finishes the whole book before answering questions about chapter one.
Common Usage Examples
from transformers import BertForSequenceClassification— fine-tune BERT for sentiment analysis- Google Search used BERT to better understand search query intent (announced 2019)
bert-base-uncasedgenerates 768-dim embeddings used for semantic search and clustering- Named entity recognition:
BertForTokenClassificationwith[B-PER, I-PER, O]label scheme - DistilBERT: 40% smaller, 60% faster, 97% of BERT's performance — widely used in production
Summary
In short: BERT proved that pre-training a large Transformer on raw text then fine-tuning on a small labelled dataset beats training from scratch on the labelled data alone — a paradigm that reshaped all of NLP.