← AI Terminology
GLUE / SuperGLUE
GLUE (General Language Understanding Evaluation) and SuperGLUE are benchmark suites for evaluating natural language understanding models across a diverse set of tasks — sentiment analysis, inference, question answering, coreference — providing a single aggregate score for comparing NLP systems.
GLUE was introduced in 2018; SuperGLUE (2019) is a harder successor after BERT saturated GLUE.
GLUE was introduced in 2018; SuperGLUE (2019) is a harder successor after BERT saturated GLUE.
Why It Matters in AI
GLUE created the NLP benchmarking culture: a single leaderboard that let researchers compare models across many tasks simultaneously. BERT surpassed human performance on GLUE within a year of GLUE's release, forcing SuperGLUE. Both benchmarks drove enormous NLP progress by providing a shared target. They are now largely saturated by frontier models, pointing the field toward harder benchmarks (MMLU, BIG-Bench, HELM).
Key Points
| Aspect | Description |
|---|---|
| Scoring | Average of task-specific metrics (accuracy, F1, Pearson r) — weighted to produce one score |
| GLUE tasks | CoLA (grammar), SST-2 (sentiment), MRPC (paraphrase), STS-B (similarity), QQP, MNLI, QNLI, RTE, WNLI |
| Saturation | GPT-3.5+ exceeds human performance on SuperGLUE — benchmark no longer differentiates frontier models |
| Successors | MMLU (knowledge breadth), BIG-Bench (emergent tasks), HELM (holistic evaluation) |
| Human baseline | GLUE human: ~87; SuperGLUE human: ~89.8 — GPT-4 exceeds both |
| SuperGLUE tasks | BoolQ, CB, COPA, MultiRC, ReCoRD, RTE, WiC, WSC — harder, require multi-sentence reasoning |
Simple Analogy
GLUE was the bar exam for NLP models — a standardised test across many legal areas (tasks). When everyone started passing it easily, SuperGLUE raised the bar. Now even that's been passed, and the field keeps creating harder exams as AI keeps graduating from each one.
Common Usage Examples
- HuggingFace
datasets.load_dataset("glue", "sst2")— loads any GLUE task - BERT fine-tuning: achieved 80.5 GLUE score (surpassing human 87 was a 2019 milestone)
glue_compute_metricsfunction in HuggingFace trainer examples — task-specific metric computation- SuperGLUE WiC (Word in Context): disambiguate word sense — requires contextual reasoning beyond surface patterns
- Kocijan et al. Winograd Schema Challenge — WSC SuperGLUE task tests commonsense coreference
Summary
In short: GLUE and SuperGLUE were the defining NLP benchmarks that drove the pre-trained Transformer revolution — both now saturated by frontier models, pointing the field toward harder challenges.