← AI Terminology
Inference-Time Scaling
Inference-time scaling is the empirical and engineering practice of improving model performance by allocating more compute at inference (samples, tokens, search depth) rather than only scaling training.
It generalises test-time compute into scaling laws and product SLAs.
It generalises test-time compute into scaling laws and product SLAs.
Why It Matters in AI
Teams must decide how quality grows with decode budget, parallel samples, and tools. Inference-time scaling frames that as a first-class curve next to parameter and data scaling — essential for reasoning products and eval design.
Key Points
| Aspect | Description |
|---|---|
| Axes | Tokens generated, number of samples, search width/depth, tools |
| Eval | Report compute used, not only model name |
| Curves | Accuracy vs FLOPs/latency/$ at inference |
| Limits | Diminishing returns; reward hacking; latency budgets |
| Related | Test-time compute, inference-time search |
| Pair with | Process rewards, verifiers, majority vote |
Simple Analogy
Not only building a bigger engine, but also choosing how long the car may race on race day — more laps of computation for a better finish.
Common Usage Examples
- Plot MMLU/GPQA vs thinking tokens
- Best-of-N scaling charts
- Arena scores at different reasoning budgets
- Cost-quality Pareto for API tiers
Summary
In short: Inference-time scaling improves answers by spending more compute at decode time — a second scaling law alongside bigger training runs.