← AI Terminology
Gradient Checkpointing
Gradient checkpointing (activation checkpointing) saves memory by storing only a subset of activations and recomputing the rest during backward.
It trades extra compute for the ability to train deeper/longer-sequence models.
It trades extra compute for the ability to train deeper/longer-sequence models.
Why It Matters in AI
Activation memory dominates transformer training. Checkpointing unlocks larger models and contexts on the same GPU at ~20–30% extra compute. Default toggle in serious fine-tunes.
Key Points
| Aspect | Description |
|---|---|
| Use | LLM full finetunes, long context training |
| APIs | torch.utils.checkpoint, HF gradient_checkpointing_enable() |
| Cost | More FLOPs, much less activation RAM |
| Idea | Keep checkpoints; recompute intermediates in backward |
| Related | Rematerialisation, activation offloading |
| Pair with | FlashAttention, mixed precision, FSDP |
Simple Analogy
Not photographing every step of a hike — only milestones — and re-walking segments if you need intermediate details later.
Common Usage Examples
model.gradient_checkpointing_enable()- Train 2× longer context without OOM
- Expect slower step time
- Combine with LoRA if still tight on memory
Summary
In short: Gradient checkpointing recomputes activations to save memory — enabling larger models and contexts at the cost of extra compute.