← AI Terminology
Bias-Variance Tradeoff
The bias-variance tradeoff is the fundamental tension in supervised learning between a model that is too simple (high bias, underfits) and one that is too complex (high variance, overfits).
Total prediction error = bias² + variance + irreducible noise.
Total prediction error = bias² + variance + irreducible noise.
Why It Matters in AI
Every modelling decision — model complexity, regularisation strength, training set size — moves you along the bias-variance spectrum. Understanding this tradeoff helps diagnose whether a model fails because it lacks the capacity to capture patterns (bias) or because it memorised the training set too closely (variance). Deep learning partially breaks the classical tradeoff at very large scales (double descent), but the concept remains foundational.
Key Points
| Aspect | Description |
|---|---|
| Bias | Error from wrong assumptions — underfitting; model systematically wrong across all data |
| Variance | Error from sensitivity to training data — overfitting; changes with different training samples |
| Complexity | Low complexity = high bias; high complexity = high variance; sweet spot = optimal generalisation |
| Irreducible | Noise inherent in the data that no model can eliminate |
| Double descent | Very large neural networks often show decreasing test error beyond the interpolation threshold — classical tradeoff breaks down |
| Regularisation | Reduces variance at cost of slight bias increase — L1/L2, dropout, early stopping |
Simple Analogy
A straight line fitted to wiggly data always misses (high bias). A polynomial that perfectly hits every training point will miss on new data (high variance). The right model complexity splits the difference — flexible enough to capture the true pattern, rigid enough not to chase noise.
Common Usage Examples
- Learning curves: plot training vs validation loss vs dataset size — divergence = high variance
- Cross-validation: estimates variance across different training subsets
- Regularisation strength
αin Ridge regression: increase α → more bias, less variance - Decision tree max_depth: shallow = high bias; deep = high variance; pruned = balanced
- Double descent observed in transformer training: test loss decreases again after interpolation threshold
Summary
In short: The bias-variance tradeoff says every model pays in one currency or the other — too simple and it's always wrong, too complex and it's only right on data it's seen before.