← AI Terminology
ROUGE Score
ROUGE (Recall-Oriented Understudy for Gisting Evaluation) is a set of automatic metrics for evaluating text summarisation and machine translation quality — measuring n-gram overlap between a machine-generated summary and reference human-written summaries.
It is the standard evaluation metric for text summarisation tasks.
It is the standard evaluation metric for text summarisation tasks.
Why It Matters in AI
Human evaluation of summaries is expensive and slow. ROUGE provides fast, reproducible, automated scores that correlate reasonably well with human judgement for factual summarisation tasks. ROUGE-1 (unigram overlap), ROUGE-2 (bigram overlap), and ROUGE-L (longest common subsequence) capture different aspects of content coverage. It remains the primary reporting metric in summarisation papers despite its limitations — it rewards lexical overlap but misses semantic equivalence ("car" ≠ "automobile" in ROUGE).
Key Points
| Aspect | Description |
|---|---|
| ROUGE-1 | Unigram (single word) recall/precision/F1 between generated and reference summary |
| ROUGE-2 | Bigram overlap — captures phrase-level similarity |
| ROUGE-L | Longest Common Subsequence — measures structural similarity while allowing gaps |
| Limitations | Misses semantic equivalence; rewards verbose outputs; doesn't penalise hallucination |
| Alternatives | BERTScore (embedding similarity), METEOR, BLEURT — learned metrics that capture semantics better |
| Recall-oriented | Original design: how much of the reference is covered — but F1 version most commonly reported |
Simple Analogy
Checking how many words from the textbook answer appear in the student's answer: a student who uses the same vocabulary as the reference answer scores well, even if they write something completely different. ROUGE is a word-overlap check — useful but not perfect at detecting meaning.
Common Usage Examples
from rouge_score import rouge_scorer; scorer = rouge_scorer.RougeScorer(['rouge1', 'rouge2', 'rougeL'])scores = scorer.score(prediction, reference)→{'rouge1': Score(precision, recall, fmeasure), ...}datasets.load_metric("rouge").compute(predictions=preds, references=refs)— HuggingFace datasets- CNN/DailyMail benchmark: ROUGE-1/2/L standard reporting for news summarisation models
- BART summarisation evaluation:
ROUGE-L F1 = 0.398on CNN/DailyMail — typical state-of-art score
Summary
In short: ROUGE measures n-gram overlap between generated and reference summaries — the standard automatic evaluation metric for text summarisation, fast and reproducible but limited by its inability to capture semantic equivalence.