← AI Terminology
PRM - Process Reward Model
A process reward model (PRM) scores intermediate reasoning steps, not only the final answer — providing denser feedback for training or search.
PRMs power step-level RL, best-of-N over partial traces, and process supervision pipelines.
PRMs power step-level RL, best-of-N over partial traces, and process supervision pipelines.
Why It Matters in AI
Final-answer rewards are sparse: a long proof is right or wrong only at the end. Step scores let trainers and search algorithms credit good partial work and catch errors early — key to math/code reasoning systems.
Key Points
| Aspect | Description |
|---|---|
| Use | Guide MCTS/beam search; RL with dense rewards |
| Risk | Mis-specified step scores → reward hacking |
| Labels | Human or automated step correctness annotations |
| Vs ORM | Outcome RM scores only the final answer |
| Pair with | Verifiers, self-consistency, outcome checks |
| Origin popularised | Uesato et al.; Lightman et al. (Let’s Verify Step by Step) |
Simple Analogy
A coach grading every move in a chess game, not only whether you won — so you learn which mid-game choices were good.
Common Usage Examples
- Math solutions scored step-by-step
- PRM-guided beam search over CoT
- Process supervision datasets for GSM8K/MATH
- Combine PRM + final-answer checker
Summary
In short: A process reward model scores each reasoning step, giving dense feedback so models can learn and search better multi-step solutions.