← AI Terminology
Distributed Training
Distributed training is the practice of splitting the work of training a neural network across multiple compute devices (GPUs, TPUs, or nodes) simultaneously, enabling models and datasets that are too large for a single device.
It is the infrastructure that makes training LLMs with billions of parameters feasible.
It is the infrastructure that makes training LLMs with billions of parameters feasible.
Why It Matters in AI
GPT-3 (175B parameters) requires ~350 GB in FP16 — far beyond any single GPU's memory. Distributed training splits this across thousands of GPUs using data, model, tensor, and pipeline parallelism in combination. Without it, frontier AI systems could not be trained. The infrastructure to coordinate distributed training — NCCL, DeepSpeed, Megatron-LM — is as important as the algorithms themselves.
Key Points
| Aspect | Description |
|---|---|
| ZeRO | DeepSpeed's Redundancy Elimination — shards optimizer states, gradients, and params across GPUs |
| Interconnect | NVLink (GPU-GPU), InfiniBand (node-node) — bandwidth determines distributed training efficiency |
| Data parallelism | Same model on each GPU; different data shard; gradients averaged — simplest form |
| Model parallelism | Model layers split across GPUs — needed when model > single GPU memory |
| Tensor parallelism | Individual weight matrices split across GPUs — Megatron-LM; for very large layers |
| Pipeline parallelism | Model stages (groups of layers) on different GPUs; micro-batches flow through the pipeline |
Simple Analogy
Building a skyscraper: one crew can't do it alone. Different crews build different floors simultaneously (pipeline parallelism); some crews work on the same floor using different blueprints (data parallelism); specialist teams handle specific structural elements (tensor parallelism). A coordinator (NCCL/NVSHMEM) keeps everyone in sync.
Common Usage Examples
torch.distributed.launch/torchrun— launch multi-GPU/multi-node training- DeepSpeed:
ds_config.jsonwith ZeRO-3 stage — trains 65B+ parameter models on consumer GPU clusters - Megatron-LM: tensor + pipeline parallelism for training GPT-scale models at NVIDIA
- HuggingFace Accelerate:
accelerate launch --multi_gpu train.py— transparent distributed training - FSDP (Fully Sharded Data Parallel): PyTorch's built-in ZeRO equivalent — shards model across GPUs
Summary
In short: Distributed training splits model training across hundreds or thousands of GPUs — the infrastructure without which LLMs and other frontier models could not be trained.