← AI Terminology
Data Parallelism
Data parallelism is a distributed training strategy where the same model is replicated across multiple devices (GPUs/TPUs), each processing a different subset of the training batch, with gradients averaged (all-reduced) across all replicas before each weight update.
It is the default distributed training approach for models that fit on a single device.
It is the default distributed training approach for models that fit on a single device.
Why It Matters in AI
Training large models on one GPU can take weeks. Data parallelism is the simplest way to scale: replicate the model, split the batch, each GPU computes gradients in parallel, average the gradients, and update — achieving near-linear throughput scaling with the number of GPUs. PyTorch DDP and JAX's
pmap implement this transparently.Key Points
| Aspect | Description |
|---|---|
| Limit | If the model exceeds single GPU memory, need model/pipeline/tensor parallelism instead |
| DP vs DDP | DataParallel (older, slower, single-node) vs DistributedDataParallel (faster, multi-node) |
| Frameworks | torch.nn.parallel.DistributedDataParallel, JAX pmap, Accelerate, DeepSpeed ZeRO |
| Model replicas | Each GPU holds a complete copy of the model — requires model to fit in single GPU memory |
| Effective batch | Global batch = per-device batch × number of devices — scale LR proportionally |
| Gradient all-reduce | After backward pass, gradients are summed/averaged across all GPUs using ring-allreduce |
Simple Analogy
A factory making 1,000 chairs with one assembly line takes 10 hours. Ten identical assembly lines each making 100 chairs cuts it to ~1 hour. Each line follows the same blueprint (model replica), each handles a subset of orders (data split), and quality checks are synchronized across all lines at the end of each shift (gradient all-reduce).
Common Usage Examples
model = torch.nn.parallel.DistributedDataParallel(model, device_ids=[local_rank])torchrun --nproc_per_node=8 train.py— launch 8-GPU DDP training- HuggingFace Accelerate:
accelerator.prepare(model, optimizer, dataloader)— handles DDP transparently - JAX:
jax.pmap(train_step)(params, batch)— vectorises training step across TPU cores - DeepSpeed ZeRO-1/2: shards optimiser states/gradients across data-parallel replicas — reduces memory per GPU
Summary
In short: Data parallelism replicates the model across GPUs and splits the data — the straightforward way to scale training when the model fits on one device.