← AI Terminology
LLM-as-Judge
LLM-as-judge uses a language model to score or compare other model outputs (quality, safety, preference) when gold labels are expensive or subjective.
It scales evaluation and RLAIF-style pipelines.
It scales evaluation and RLAIF-style pipelines.
Why It Matters in AI
Human eval does not scale; exact-match metrics miss open-ended quality. Judge models enable large offline evals and CI gates — if calibrated carefully against humans and debiased.
Key Points
| Aspect | Description |
|---|---|
| Use | Chat quality, summarisation, safety filters |
| Modes | Pointwise scores, pairwise prefs, rubrics |
| Risks | Position bias, verbosity bias, self-preference |
| Tools | MT-Bench style judges; proprietary eval APIs |
| Related | Golden datasets, RLAIF, HELM |
| Mitigations | Swap order, rubrics, ensembles, human calibration |
Simple Analogy
Hiring a senior reviewer to grade junior essays with a rubric when you cannot read ten thousand papers yourself.
Common Usage Examples
- Pairwise: which answer is more helpful?
- Rubric scores 1–5 on faithfulness
- Calibrate judge vs human kappa
- CI fails if judge win-rate drops
Summary
In short: LLM-as-judge scores other model outputs with an LLM — scalable eval that must be calibrated against human judgment.