← AI Terminology
Gradient Accumulation
Gradient accumulation sums gradients over multiple micro-batches before applying an optimiser step, simulating a larger effective batch size under memory limits.
It is essential for training large models on limited GPUs.
It is essential for training large models on limited GPUs.
Why It Matters in AI
Desired batch sizes exceed VRAM. Accumulation achieves the same average gradient with serial micro-batches. Every LLM fine-tune guide uses
grad_accum_steps.Key Points
| Aspect | Description |
|---|---|
| Impl | loss.backward() several times; step/zero every N |
| Norm | Scale loss by 1/N or average correctly |
| Sync | In DDP, often sync only on step boundaries |
| Related | Mixed precision, FSDP |
| Tradeoff | Same memory as micro-batch; slower wall-clock than true big batch |
| Effective batch | micro_batch × accum_steps × #GPUs |
Simple Analogy
Paying a large bill with several smaller transfers that add up to the same total before the bank posts the payment.
Common Usage Examples
- HF Trainer
gradient_accumulation_steps=8 - Effective batch 256 with micro 2 on 16 GPUs
- Combine with gradient checkpointing
- Log optimiser steps ≠ micro-batch steps
Summary
In short: Gradient accumulation builds a large effective batch from many small backward passes — training big batches when memory is tight.