← AI Terminology
Pass@k
Pass@k is an evaluation metric for code (and similar) generation: the probability that at least one of k sampled solutions passes all unit tests.
It is standard for HumanEval-style coding benchmarks.
It is standard for HumanEval-style coding benchmarks.
Why It Matters in AI
A single sample underestimates models that can solve problems with multiple tries. Pass@k captures sampling-based reliability important for coding agents and best-of-N systems. Always report k and temperature.
Key Points
| Aspect | Description |
|---|---|
| Use | HumanEval, MBPP, LiveCodeBench |
| Caveat | High k ≠ good interactive UX if pass@1 poor |
| Common | pass@1, pass@10, pass@100 |
| Related | Best-of-N, functional correctness |
| Definition | Chance ≥1 of k samples is fully correct |
| Unbiased est. | Chen et al. estimator from n samples |
Simple Analogy
Giving a student k attempts at a lab exam and counting success if any attempt works — measures potential with retries.
Common Usage Examples
- HumanEval pass@1 vs pass@10 tables
- Sample n=20 to estimate pass@k
- Temperature sweeps for coding models
- Agent loops that effectively raise k
Summary
In short: Pass@k estimates the chance that any of k samples is correct — the standard coding-generation metric under test suites.