← AI Terminology
Regularization
Regularization is any technique that reduces overfitting by constraining model complexity — adding a penalty for large weights, randomly disabling neurons, stopping training early, or augmenting training data, so the model learns generalisable patterns rather than memorising noise.
It is the collective term for all methods that trade training accuracy for generalisation.
It is the collective term for all methods that trade training accuracy for generalisation.
Why It Matters in AI
Without regularization, powerful models memorise their training data: they fit every noise sample and quirk, producing high training accuracy but poor generalisation. Regularization forces the model to find simpler, more robust solutions. It is not one technique but a family: weight decay (penalise large weights), dropout (randomly disable neurons), data augmentation (artificially diversify training data), early stopping (stop before overfitting deepens), batch normalisation (reduces internal covariate shift). All modern deep learning pipelines use multiple regularization methods simultaneously.
Key Points
| Aspect | Description |
|---|---|
| Dropout | Randomly zero activations during training — forces redundant representations |
| Batch norm | Normalises layer outputs — reduces internal covariate shift; mild regularization effect |
| Early stopping | Monitor validation loss; stop training when it stops improving — simplest regularization |
| Label smoothing | Soften hard labels (0 → 0.1, 1 → 0.9) — prevents overconfidence on training labels |
| Data augmentation | Random flips, crops, noise — increases effective dataset diversity without new labels |
| L1/L2 (weight decay) | Add weight magnitude penalty to loss — reduces weight sizes; L2 most common in neural nets |
Simple Analogy
Study habits that prevent cramming: instead of memorising every past exam paper (overfitting), a student takes practice exams under time pressure (early stopping), covers a diverse range of topics (data augmentation), and uses flashcard review rather than word-for-word memorisation (weight decay). They learn the subject, not the specific past papers.
Common Usage Examples
AdamW(model.parameters(), weight_decay=0.01)— L2 regularization via weight decaynn.Dropout(0.5)inserted between fully connected layers — 50% neuron dropout- ImageNet training:
transforms.RandomHorizontalFlip() + transforms.RandomCrop(224)— data augmentation EarlyStopping(monitor='val_loss', patience=5, restore_best_weights=True)— Keras callbackCrossEntropyLoss(label_smoothing=0.1)— label smoothing regularization in PyTorch
Summary
In short: Regularization is the family of techniques that prevent overfitting by constraining model complexity — every production deep learning pipeline uses multiple regularization methods simultaneously to ensure the model generalises rather than memorises.