← AI Terminology
MMLU - Massive Multitask Language Understanding
MMLU (Massive Multitask Language Understanding) is a benchmark that tests language model knowledge across 57 academic subjects — including STEM, humanities, social sciences, law, and medicine — using multiple-choice questions drawn from real educational exams.
It became the standard measure of LLM broad knowledge and reasoning capability after its 2020 release.
It became the standard measure of LLM broad knowledge and reasoning capability after its 2020 release.
Why It Matters in AI
Before MMLU, LLM evaluation was fragmented across narrow task-specific benchmarks. MMLU provided a unified, broad signal: if a model scores 90% across 57 subjects, it has demonstrated genuine academic-level knowledge in fields ranging from clinical medicine to abstract algebra. It drove public awareness of the rapid capability progression of GPT-3 → GPT-4 → frontier models (human expert performance ≈ 89%; GPT-4 ≈ 86%; Claude 3 Opus ≈ 88%). MMLU is now routinely cited in every major model announcement.
Key Points
| Aspect | Description |
|---|---|
| Scale | ~14,000 questions across subjects — balanced, not weighted by subject popularity |
| Format | 4-choice multiple-choice questions — model outputs A/B/C/D |
| MMLU-Pro | Harder variant (2024) with 10-choice questions and more reasoning-heavy problems |
| Subjects | 57 subjects: STEM, law, ethics, history, economics, medicine, computer science, and more |
| Limitations | Multiple-choice format limits depth; questions may appear in training data (contamination risk) |
| Human baseline | Average human: ~34%; non-expert human: ~55%; expert human: ~89% |
Simple Analogy
A comprehensive university entrance exam across every department: not just one subject but law, medicine, physics, history, and ethics in one sitting. A student (LLM) who scores 90% across all departments has demonstrated genuinely broad academic knowledge — not just a specialist.
Common Usage Examples
lm_eval --model hf --model_args pretrained=meta-llama/Llama-3-8B --tasks mmlu— EleutherAI eval harness- Model cards: every frontier model release includes MMLU score (5-shot standard evaluation)
datasets.load_dataset("cais/mmlu", "all")— HuggingFace MMLU dataset- GPT-4 technical report: MMLU 86.4% (5-shot) — first model approaching expert human performance
- MMLU-Pro:
datasets.load_dataset("TIGER-Lab/MMLU-Pro")— harder 10-choice variant
Summary
In short: MMLU tests language model knowledge across 57 academic subjects — the standard benchmark that revealed the rapid capability progression of frontier LLMs and remains the primary measure of broad academic knowledge.