← AI Terminology
GRPO - Group Relative Policy Optimization
GRPO (Group Relative Policy Optimization) is a reinforcement learning algorithm that updates a policy using relative advantages among a group of sampled outputs for the same prompt — without a separate critic/value network in the DeepSeek-style formulation.
It is widely associated with DeepSeek-R1/math RL training.
It is widely associated with DeepSeek-R1/math RL training.
Why It Matters in AI
PPO needs a value model and careful KL control; GRPO simplifies on-policy LLM RL by normalising rewards within a prompt’s sample group. It became a practical recipe for open reasoning-model training.
Key Points
| Aspect | Description |
|---|---|
| Use | Math/code RL with verifiable rewards |
| Idea | Sample a group per prompt; compute relative advantages inside the group |
| Origin | DeepSeekMath / DeepSeek-R1 training stack |
| Benefit | Simpler, memory-friendlier RL for LLMs |
| Related | PPO, RLOO, REINFORCE with baseline |
| No critic | DeepSeek-style GRPO avoids a learned value network |
Simple Analogy
Grading essays on a curve within each class section — scores matter relative to peers who answered the same question, not on an absolute universal scale alone.
Common Usage Examples
- DeepSeek-R1 technical report RL section
- Open-source GRPO trainers (TRL/community)
- Group size hyperparameters in RL configs
- Compare PPO vs GRPO memory on 7B–70B
Summary
In short: GRPO optimises a policy using relative scores among grouped samples per prompt — a simpler RL method popularised for training open reasoning models.