← AI Terminology
GAE - Generalized Advantage Estimation
GAE (Generalized Advantage Estimation) computes a exponentially weighted mix of multi-step TD advantages to trade bias and variance in advantage estimates.
Introduced with TRPO-era policy gradients; standard in PPO implementations.
Introduced with TRPO-era policy gradients; standard in PPO implementations.
Why It Matters in AI
Advantage estimation quality makes or breaks on-policy RL. GAE’s λ parameter is a ubiquitous hyperparameter in PPO training for robots and games — practical RL engineering literacy.
Key Points
| Aspect | Description |
|---|---|
| λ | 0 → low variance high bias; 1 → Monte Carlo-like |
| Paper | Schulman et al., GAE (2015) |
| Formula | Weighted sum of TD residuals with λ |
| Related | Advantage function, TD(λ) |
| Used in | PPO, A2C-style algorithms everywhere |
| Practice | λ≈0.9–0.98 common |
Simple Analogy
Blending short-term and long-term performance reviews with a fading memory weight — not only yesterday, not only the whole career.
Common Usage Examples
- PPO configs:
gae_lambda=0.95 - Ablate λ vs learning stability
- Implement GAE on trajectory buffers
- Pair with value function baselines
Summary
In short: GAE estimates advantages with a bias-variance knob λ — the standard advantage estimator inside PPO-style deep RL.