← AI Terminology
Calibration
Calibration is the degree to which a model's predicted probabilities match actual outcome frequencies — a perfectly calibrated model that says "70% chance of rain" should be right 70% of the time it says that.
It measures whether probability outputs are trustworthy, not just directionally correct.
It measures whether probability outputs are trustworthy, not just directionally correct.
Why It Matters in AI
High accuracy does not imply good calibration. A model that predicts 0.99 probability for every positive and is correct 80% of the time is wildly overconfident. In medical diagnosis, fraud detection, and financial risk, decision-makers use the probability directly — not just the label — so calibration errors propagate into bad decisions. Calibration is also critical for LLM uncertainty: a model that confidently states falsehoods is poorly calibrated.
Key Points
| Aspect | Description |
|---|---|
| ECE | Expected Calibration Error: weighted average of reliability curve deviation from diagonal |
| Platt scaling | Fit a logistic regression on validation set outputs to recalibrate probabilities |
| Overconfidence | Most neural networks are overconfident — predicted probability > actual frequency |
| LLM calibration | LLMs often confidently hallucinate — calibration alignment is an active safety research area |
| Reliability curve | Plot predicted probability (x) vs observed frequency (y) — perfect calibration is the diagonal |
| Temperature scaling | Post-hoc calibration: divide logits by T > 1 to soften probabilities without retraining |
Simple Analogy
A confident doctor who says "I'm 95% sure it's benign" should be right 95 times out of 100. If they're only right 70 times, they're overconfident — their probability numbers can't be taken at face value. Calibration is the check on whether confidence scores actually track reality.
Common Usage Examples
sklearn.calibration.calibration_curve(y_true, y_prob)— plot the reliability diagramCalibratedClassifierCV(base_estimator, method='isotonic')— isotonic regression recalibration- Temperature scaling:
logits_calibrated = logits / Twhere T found on validation set sklearn.metrics.brier_score_lossandlog_loss— both sensitive to calibration- DeepMind calibration paper (Guo et al., 2017): modern neural networks are badly overconfident
Summary
In short: Calibration is whether a model's confidence scores mean what they say — a model that says "80% confident" should be right 80% of the time, not 95% or 60%.