← AI Terminology
Label Smoothing
Label smoothing replaces hard one-hot targets with softened distributions (e.g. 0.9 on the correct class, small mass on others) to regularise classification training.
Common in vision and some sequence models.
Common in vision and some sequence models.
Why It Matters in AI
Hard labels encourage overconfident logits. Smoothing improves calibration and generalisation in many classification settings and appears in Transformer training recipes (including translation).
Key Points
| Aspect | Description |
|---|---|
| Use | ImageNet training, some seq2seq |
| Caveat | Can hurt if you need sharp probabilities |
| Effect | Less overconfidence; regularisation |
| Smooth | (1−ε) on correct + ε/(K−1) on others |
| Related | Knowledge distillation soft labels, Mixup |
| Hard target | 1 on correct class, 0 elsewhere |
Simple Analogy
Instead of saying the answer is 100% definitively cat, the teacher says “mostly cat, leave a little room for doubt” — softer supervision.
Common Usage Examples
CrossEntropyLoss(label_smoothing=0.1)- Vision recipes with ε=0.1
- Calibration plots before/after
- Disable when training teacher for hard distillation carefully
Summary
In short: Label smoothing softens one-hot targets to reduce overconfidence — a simple regulariser in classification training.