← AI Terminology
SWE-bench
SWE-bench evaluates models on real GitHub issues: given a codebase and bug report, produce a patch that passes the project’s tests.
It became the flagship benchmark for software-engineering agents.
It became the flagship benchmark for software-engineering agents.
Why It Matters in AI
Toy HumanEval problems understate real engineering. SWE-bench measures repository-level repair — closer to developer copilots and autonomous coding agents. Variants (Verified, Multimodal, Live) track progress.
Key Points
| Aspect | Description |
|---|---|
| Hard | Long context, navigation, multi-file reasoning |
| Task | Issue → edit files → pass tests |
| Agents | Scaffolding often matters as much as base model |
| Origin | Princeton/Chicago et al. SWE-bench |
| Related | pass@k, LiveCodeBench |
| Variants | SWE-bench Verified, lite splits |
Simple Analogy
Not writing a function on a blank page, but fixing a real bug in a large unfamiliar repo until CI goes green.
Common Usage Examples
- SWE-bench leaderboards
- Agent scaffolds (tools, retrieval over repo)
- SWE-bench Verified for cleaner labels
- Track resolve rates over model releases
Summary
In short: SWE-bench tests whether AI can fix real GitHub issues with patches that pass tests — the premier software-engineering agent benchmark.