← AI Terminology
HumanEval
HumanEval is a benchmark from OpenAI (Chen et al., 2021) consisting of 164 programming problems — each with a function signature, docstring, and unit tests — used to evaluate LLMs' ability to generate correct Python code from natural language descriptions.
It is the most widely cited benchmark for code generation capability.
It is the most widely cited benchmark for code generation capability.
Why It Matters in AI
HumanEval established the standard for measuring coding AI: generate a function body, run the unit tests, report pass@k (pass rate from k generated samples). It enabled direct comparison across Codex, GitHub Copilot, GPT-4, Claude, and open-source models. Progress has been rapid: original Codex scored 28.8% pass@1; Claude 3.5 Sonnet scores ~92% — a signal of how dramatically coding AI has advanced in 3 years.
Key Points
| Aspect | Description |
|---|---|
| Format | 164 functions: docstring + signature given; body generated by the model; unit tests evaluate |
| pass@k | Run k samples per problem; pass = any sample passes all tests; reported as pass@1, pass@10, pass@100 |
| Difficulty | Range from trivial string manipulation to tricky algorithmic reasoning |
| Successors | SWE-bench (real GitHub issues), MBPP, BigCodeBench, LiveCodeBench (contamination-resistant) |
| Limitations | 164 problems is small; single-function; no multi-file, no imports, mostly algorithmic |
| Contamination | Models trained post-2021 may have seen HumanEval problems — inflated scores for some models |
Simple Analogy
HumanEval is a standardised coding interview: 164 questions with automatic unit test graders. A model that passes more test cases is a better programmer — just like an interview candidate who correctly solves more coding problems in a timed test. The interviewer is automated and consistent.
Common Usage Examples
evaluate.load("openai_humaneval"); results = evaluate.compute(predictions=[...], references=[...])- Codex (code-davinci-002): 28.8% pass@1 — the 2021 baseline that launched code LLM research
- GPT-4: ~67% pass@1; Claude 3.5 Sonnet: ~92% pass@1 (2024 figures)
- EvalPlus: extended HumanEval with more test cases per problem — catches solutions that pass original but are wrong
- LiveCodeBench: fresh competitive programming problems — resists contamination from training data
Summary
In short: HumanEval is the standard benchmark for code generation — 164 Python problems evaluated by unit tests, where pass@1 has gone from 28% (Codex, 2021) to 90%+ (frontier models, 2024–25).