← AI Terminology
BLEU Score - Bilingual Evaluation Understudy
BLEU stands for Bilingual Evaluation Understudy: an automatic metric for evaluating machine translation quality by measuring the overlap of n-grams between a model's output and one or more human reference translations.
Introduced by Papineni et al. (2002) at IBM, it became the standard automatic MT evaluation metric.
Introduced by Papineni et al. (2002) at IBM, it became the standard automatic MT evaluation metric.
Why It Matters in AI
BLEU enabled large-scale, cost-effective evaluation of translation systems without human annotators for every experiment. It correlated well with human judgement in the era of statistical MT. However, neural MT systems have exposed its limits — a model can produce a fluent, accurate translation that scores low because it uses different but equally valid words. BLEU is still widely reported but is increasingly complemented by METEOR, chrF, and COMET.
Key Points
| Aspect | Description |
|---|---|
| Range | 0–100 (or 0–1); 50+ is considered high quality; state-of-the-art MT scores ~45–55 on WMT |
| Beyond MT | BLEU used in image captioning, code generation, summarisation — often misleadingly |
| Limitations | Does not capture fluency, meaning, or paraphrase — two sentences can be equivalent with 0 overlap |
| Alternatives | METEOR (stems, synonyms), chrF (character n-grams), COMET (neural, correlates better with human judgement) |
| Brevity penalty | Penalises outputs shorter than the reference — prevents trivially high precision via one word |
| n-gram precision | Fraction of model's n-grams (1–4 words) that appear in the reference — averaged across n=1..4 |
Simple Analogy
BLEU is like grading an essay by counting how many exact words and phrases match a model answer key. A student who writes an equally correct but differently worded answer gets penalised. The score measures lexical similarity, not quality.
Common Usage Examples
sacrebleu.corpus_bleu(hypotheses, [references])— the standard Python BLEU implementation- WMT shared task: competing MT systems reported in BLEU, now increasingly in COMET
evaluate.load("bleu"); metric.compute(predictions=[...], references=[[...]])in HuggingFace- BLEU commonly reported in seq2seq fine-tuning papers even when ROUGE would be more appropriate
- SacreBLEU introduced to standardise tokenisation — different tokenisers gave incomparable BLEU scores
Summary
In short: BLEU measures how much a model's text overlaps with a human reference — fast and cheap, but poor at capturing meaning when word choice differs from the reference.