← AI Terminology
Brier Score
The Brier score is a metric that measures the accuracy of probabilistic predictions by computing the mean squared error between predicted probabilities and the actual binary outcomes.
Lower is better: 0 is perfect, 0.25 is the score of a model that always predicts 0.5, and 1.0 is maximally wrong.
Lower is better: 0 is perfect, 0.25 is the score of a model that always predicts 0.5, and 1.0 is maximally wrong.
Why It Matters in AI
Unlike accuracy (which only cares about the hard label decision), the Brier score rewards models that are both correct and confident when correct, and penalises overconfident wrong predictions. It is a proper scoring rule — meaning a model that knows the true probabilities will minimise its expected Brier score by reporting them honestly. This makes it ideal for evaluating weather forecasts, clinical risk models, and any application where calibrated probabilities matter.
Key Points
| Aspect | Description |
|---|---|
| Range | 0 (perfect) to 1 (maximally wrong) — uninformative baseline (always predict base rate) ≈ p(1−p) |
| Formula | BS = (1/n) Σ (p_i − o_i)² where p = predicted probability, o = actual outcome (0/1) |
| Multiclass | Extended to Brier score across all class probabilities |
| vs Log-loss | Log-loss is also proper and more sensitive to extreme wrong probabilities; Brier is more interpretable |
| Decomposition | Reliability (calibration) + Resolution (sharpness) + Uncertainty — diagnoses error sources |
| Proper scoring | Honest probability reporting minimises expected Brier score — no incentive to game |
Simple Analogy
A weather forecaster who says "70% chance of rain" and it rains scores better than one who says "50%" — but both score better than one who says "95%" and it doesn't rain. Brier score rewards confident correct forecasts and punishes confident wrong ones, in proportion to how confident and how wrong.
Common Usage Examples
sklearn.metrics.brier_score_loss(y_true, y_prob)— binary Brier score in scikit-learn- Clinical risk models (e.g. 30-day mortality prediction): Brier score standard evaluation metric
- Weather forecasting accuracy comparison: ECMWF vs national weather services scored by Brier score
- Evaluating a churn model's probability outputs, not just the hard 0/1 classification
- Brier Skill Score:
BSS = 1 − BS/BS_reference— compares model to a climatological baseline
Summary
In short: The Brier score measures probabilistic prediction quality — it rewards calibrated confidence and punishes overconfident mistakes, making it the right metric when probabilities matter.