← AI Terminology
L1/L2 Regularization
L1 and L2 regularization are techniques that add a penalty term to the loss function proportional to the magnitude of model weights — discouraging large weights to prevent overfitting and improve generalisation.
L1 (Lasso) penalises the sum of absolute weight values; L2 (Ridge) penalises the sum of squared weight values.
L1 (Lasso) penalises the sum of absolute weight values; L2 (Ridge) penalises the sum of squared weight values.
Why It Matters in AI
Overfitting occurs when a model memorises training data rather than learning general patterns — large weights are a symptom. Regularization constrains weight growth, forcing the model to find simpler solutions. L2 is the default in most neural network training (via weight decay); L1 induces sparsity, zeroing out less-useful weights and enabling feature selection. Both are fundamental tools for improving model robustness on held-out data.
Key Points
| Aspect | Description |
|---|---|
| L1 (Lasso) | Penalty = λ × Σ |
| L2 (Ridge) | Penalty = λ × Σwᵢ² — shrinks all weights towards zero; rarely produces exact zeros |
| Elastic Net | L1 + L2 combined — balances sparsity (L1) with stability (L2); common in linear models |
| λ (lambda) | Regularization strength hyperparameter — larger λ → stronger shrinkage → simpler model |
| Weight decay | L2 regularization implemented as weight decay in SGD/Adam — optimizer = Adam(weight_decay=1e-4) |
| Neural networks | L2 (weight decay) is standard; L1 used less often due to non-differentiability at zero |
Simple Analogy
A penalty system for packing luggage: L2 charges proportional to the square of each item's weight (heavy items cost disproportionately), so everything stays small; L1 charges a flat fee per item, so you end up removing some items entirely (sparsity) rather than making everything slightly smaller.
Common Usage Examples
torch.optim.AdamW(model.parameters(), weight_decay=1e-4)— L2 regularization via AdamW- Scikit-learn Ridge:
Ridge(alpha=1.0)— L2 regularized linear regression - Scikit-learn Lasso:
Lasso(alpha=0.01)— L1 for feature selection in linear models keras.layers.Dense(64, kernel_regularizer=keras.regularizers.l2(1e-4))— per-layer L2- Elastic Net:
ElasticNet(alpha=0.5, l1_ratio=0.5)— mix of L1 and L2
Summary
In short: L1 and L2 regularization prevent overfitting by penalising large weights — L2 (weight decay) shrinks all weights smoothly and is the default in neural network training; L1 zeroes out weak weights and enables feature selection.