← AI Terminology
DeepSpeed ZeRO
ZeRO (Zero Redundancy Optimizer) in DeepSpeed partitions optimiser states, gradients, and parameters across devices so each GPU holds only a slice — enabling much larger models.
Stages 1–3 progressively shard more state.
Stages 1–3 progressively shard more state.
Why It Matters in AI
ZeRO made multi-billion-parameter training practical on commodity clusters and influenced FSDP. Choosing ZeRO stage vs tensor/pipeline parallel is core large-scale training design.
Key Points
| Aspect | Description |
|---|---|
| Origin | Microsoft DeepSpeed team |
| Offload | CPU/NVMe offload options for more savings |
| Stage 1 | Shard optimiser states |
| Stage 2 | + shard gradients |
| Stage 3 | + shard parameters (most memory savings) |
| Vs FSDP | Similar goals; different stack/ecosystem |
Simple Analogy
A moving company where no truck carries the entire house’s inventory — furniture split across trucks, reassembled room by room when needed.
Common Usage Examples
deepspeed --zero_stage 3- ZeRO-Offload for single-node large models
- HF + DeepSpeed integration configs
- Pick stage by memory profiling
Summary
In short: DeepSpeed ZeRO shards training state across GPUs in stages — a foundational technology for training large models under memory limits.