← AI Terminology
Generalization
Generalization is the ability of a trained model to perform well on new, unseen data that it was not exposed to during training — the fundamental goal of machine learning.
A model that only performs well on its training data has memorised rather than learned.
A model that only performs well on its training data has memorised rather than learned.
Why It Matters in AI
Every ML system is ultimately deployed on data it hasn't seen. Generalization is the gap between training performance and real-world performance — closing it is the central challenge of the field. Poor generalization (overfitting) produces impressive benchmark numbers that collapse in production. Understanding generalization has led to foundational techniques: regularisation, cross-validation, dropout, data augmentation, and scaling laws.
Key Points
| Aspect | Description |
|---|---|
| Overfitting | High training accuracy, low test accuracy — model memorised training data |
| Underfitting | Low training AND test accuracy — model too simple to capture the pattern |
| Double descent | Very large models can generalise despite interpolating training data — classical theory incomplete |
| Regularisation | L1/L2, dropout, early stopping, data augmentation — all techniques for improving generalisation |
| Training vs test | Training accuracy measures memorisation; test accuracy measures generalisation |
| Distribution shift | Test data from a different distribution than training data — even well-generalising models fail |
Simple Analogy
A student who memorises exam answers without understanding the material gets 100% on the practice test and fails the real exam (poor generalisation). A student who understands the underlying concepts can answer novel variations they've never seen (good generalisation). The practice test is training data; the real exam is the test set.
Common Usage Examples
- Train/validation/test split: evaluate generalisation on held-out test set never touched during training
- Learning curves: plot training and validation loss vs dataset size — convergence indicates good generalisation
- Out-of-domain evaluation: test a medical model trained on US hospitals on data from African clinics
- Weight decay:
AdamW(weight_decay=0.01)— L2 regularisation that improves generalisation - Generalisation benchmark: WILDS dataset — explicitly tests out-of-distribution generalisation for many domains
Summary
In short: Generalization is whether a model has learned the pattern or memorised the examples — the difference between a model that works in the lab and one that works in production.