← AI Terminology
FSDP - Fully Sharded Data Parallel
FSDP (Fully Sharded Data Parallel) shards model parameters, gradients, and optimiser states across data-parallel workers, gathering weights only as needed for compute.
It is PyTorch’s standard path to train large models across multi-GPU servers.
It is PyTorch’s standard path to train large models across multi-GPU servers.
Why It Matters in AI
Data parallel alone replicates the full model on every GPU — impossible at 70B+. FSDP shards state like ZeRO-3, enabling large-model training in native PyTorch. Essential distributed-training literacy.
Key Points
| Aspect | Description |
|---|---|
| Use | LLM full finetune and pretrain on multi-GPU |
| Shards | Params, grads, optimiser states across ranks |
| Vs DDP | DDP replicates full model; FSDP shards |
| PyTorch | torch.distributed.fsdp.FullyShardedDataParallel |
| Related | DeepSpeed ZeRO stages, tensor parallel |
| Tradeoff | Communication vs memory savings |
Simple Analogy
Instead of every chef keeping a full copy of a huge recipe book, pages are split among chefs and borrowed only when cooking that section.
Common Usage Examples
- Wrap model in FSDP for multi-GPU finetune
- Combine with activation checkpointing
- Compare memory to DDP OOM
- HF Accelerate FSDP config
Summary
In short: FSDP shards model and optimiser state across GPUs — PyTorch’s primary way to data-parallel train models too large to replicate.