← AI Terminology
Overfitting
Overfitting is when a model learns the training data too well — including its noise and irrelevant patterns — resulting in high training accuracy but poor performance on unseen data, because the model memorised rather than generalised.
It is the most common failure mode in supervised machine learning.
It is the most common failure mode in supervised machine learning.
Why It Matters in AI
A model with 100% training accuracy and 60% test accuracy is useless for production — it has memorised the training set, not learned the underlying task. Overfitting is especially common with large models, small datasets, or long training runs. Every regularisation technique (dropout, weight decay, data augmentation, early stopping) exists primarily to combat overfitting. Understanding and detecting it via the train/validation loss gap is fundamental to ML practice.
Key Points
| Aspect | Description |
|---|---|
| LLMs | Large language models rarely classically overfit due to scale — but memorise training data |
| Cause | Model capacity too high for dataset size; insufficient regularisation; too many epochs |
| Symptom | Train loss decreases but validation loss plateaus or increases — the "divergence" in training curves |
| Underfitting | Opposite: model too simple, doesn't learn training data well — high bias, low variance |
| Regularisation | Dropout, weight decay, L1/L2, data augmentation, early stopping — all reduce overfitting |
| Bias-variance tradeoff | Overfitting = high variance (sensitive to training set); underfitting = high bias (misses patterns) |
Simple Analogy
A student who memorises every past exam question word-for-word: they score 100% on practice tests but fail the real exam when questions are phrased differently. They memorised patterns specific to past exams (noise) rather than understanding the subject (generalisation).
Common Usage Examples
- Detecting: plot
train_lossvsval_loss— if they diverge (val increases while train decreases), overfitting nn.Dropout(0.5)— randomly zero 50% of activations during training — strong regularisationEarlyStopping(monitor='val_loss', patience=5)— stop before overfitting deepensweight_decay=1e-4in AdamW — L2 regularisation to penalise large weights- Data augmentation: random crops, flips, colour jitter — effectively increases dataset size
Summary
In short: Overfitting means the model memorised its training data rather than learning general patterns — the most common ML failure mode, detected by a widening gap between training and validation performance, and addressed by regularisation.