← AI Terminology
Model Parallelism
Model parallelism is a distributed training strategy that partitions a model's layers or parameters across multiple GPUs — enabling training of models too large to fit on a single device — as opposed to data parallelism, which replicates the model and splits the data.
Large-scale forms include tensor parallelism (split within a layer) and pipeline parallelism (split across layers).
Large-scale forms include tensor parallelism (split within a layer) and pipeline parallelism (split across layers).
Why It Matters in AI
A single A100 GPU holds ~80GB. Llama 3 70B in FP16 requires ~140GB for weights alone, before gradients or activations. No single GPU can train it. Model parallelism splits the model across multiple GPUs, with each holding and computing only its shard. 3D parallelism (data + tensor + pipeline parallelism) is how frontier labs train 100B+ parameter models across thousands of GPUs.
Key Points
| Aspect | Description |
|---|---|
| FSDP / ZeRO | Shard parameters, gradients, and optimizer state across GPUs — all GPUs see all data |
| Megatron-LM | NVIDIA's framework implementing 3D parallelism for transformer training |
| 3D parallelism | Combine data + tensor + pipeline — how GPT-4, Llama large-scale training works |
| Tensor parallelism | Split individual weight matrices across GPUs (e.g. split attention heads) — requires fast inter-GPU bandwidth |
| NVLink / InfiniBand | High-bandwidth GPU interconnects critical for tensor parallelism — bandwidth bottleneck |
| Pipeline parallelism | Assign groups of layers to each GPU — GPUs process different mini-batches in a pipeline |
Simple Analogy
Building a skyscraper with a crew too large for one floor: split the crew across floors (pipeline), let each crew member handle one section of each floor (tensor), and rotate floors to different crews as each completes (data parallelism). No single worker handles everything, but together they build faster than any one crew could.
Common Usage Examples
- FSDP:
FullyShardedDataParallel(model, auto_wrap_policy=...)— PyTorch native ZeRO-3 equivalent - Tensor parallel:
megatron.core.tensor_parallel.ColumnParallelLinear(in_features, out_features) - DeepSpeed ZeRO-3:
"zero_optimization": {"stage": 3}— shards params + gradients + optimizer state accelerate launch --num_processes 8 --multi_gpu train.py— multi-GPU training with Accelerate- Pipeline:
torch.distributed.pipeline.sync.Pipe(model, chunks=8)— pipeline parallel 8 micro-batches
Summary
In short: Model parallelism splits a model's parameters across multiple GPUs to enable training models too large for a single device — the essential technique for training 70B+ parameter models at frontier scale.