← AI Terminology
Bias (in Evaluation)
Evaluation bias refers to systematic errors in how a model's performance is measured — including data leakage, benchmark contamination, improper test set splits, and cherry-picked metrics — that cause reported performance to be better than real-world performance.
It is distinct from model bias (unfair predictions) and statistical bias (systematic underestimation of a parameter).
It is distinct from model bias (unfair predictions) and statistical bias (systematic underestimation of a parameter).
Why It Matters in AI
Evaluation bias is how impressive benchmark numbers fail to translate into product success. A model that achieves 95% on a contaminated benchmark, or a test set that overlaps training data, is reporting an inflated score. As LLMs are trained on web-scale data that likely includes many benchmarks, contamination is a pervasive problem — making rigorous evaluation one of the hardest open problems in AI.
Key Points
| Aspect | Description |
|---|---|
| Mitigation | Hold-out test sets never touched during development; dynamic benchmarks; human evaluation |
| Metric bias | Choosing the metric that makes the model look best; ignoring failure modes |
| Data leakage | Test data (or correlated data) appears in training — inflates measured performance |
| Selection bias | Test set not representative of real use cases — high benchmark ≠ real-world usefulness |
| Evaluation on train | Accidentally evaluating on training data — worst form of leakage |
| Benchmark contamination | Model trained on data containing benchmark questions — common for LLMs trained on the web |
Simple Analogy
A student who steals tomorrow's exam questions scores 100% — that doesn't mean they learned the material. Evaluation bias in AI is the same: the score is real, but the learning it supposedly reflects is not. The test was compromised before it was taken.
Common Usage Examples
- LLM benchmark contamination: GPT-4 likely saw MMLU questions during pre-training — scores disputed
- Data leakage: using
StandardScaler.fit_transform(X)on the whole dataset before train/test split sklearn.model_selection.train_test_split— always split before any preprocessing to avoid leakage- LMSYS Arena as a contamination-resistant benchmark: human preferences on novel prompts not in training
- ARC Prize (ARC-AGI): held-out test set never released publicly to prevent contamination
Summary
In short: Evaluation bias makes models look better than they are — the most common forms are data leakage and benchmark contamination, and both are widespread in LLM evaluation.