← AI Terminology
Pipeline Parallelism
Pipeline parallelism is a distributed training strategy that partitions a model's layers across multiple GPUs in a sequential pipeline — each GPU holds a subset of layers and processes a different micro-batch simultaneously, keeping all GPUs busy like an assembly line.
It is one of the three axes of 3D parallelism (with data and tensor parallelism) used for training frontier LLMs.
It is one of the three axes of 3D parallelism (with data and tensor parallelism) used for training frontier LLMs.
Why It Matters in AI
A single GPU cannot hold a 70B-parameter model's layers even in FP16. Tensor parallelism splits within layers (requiring fast NVLink between GPUs on the same node); pipeline parallelism splits across layers (can span nodes over slower InfiniBand). For cluster-scale training (1,000+ GPUs), pipeline parallelism distributes layers across nodes, enabling models of arbitrary depth. The challenge is "pipeline bubbles" — GPU idle time waiting for earlier stages to complete.
Key Points
| Aspect | Description |
|---|---|
| Bandwidth | Activations pass between pipeline stages — requires only send/recv (lower bandwidth than tensor parallel) |
| Megatron-LM | Implements interleaved 1F1B pipeline parallelism — used for GPT-3, Llama, Falcon training |
| 1F1B schedule | One Forward, One Backward — interleaves forward/backward micro-batches to minimise idle time |
| Micro-batches | Mini-batch split into K micro-batches — pipeline stages overlap by processing different micro-batches |
| Pipeline bubble | Idle time at start (fill) and end (drain) of pipeline — reduces efficiency by ~1/K fraction |
| Layer assignment | GPU 0: layers 0–7; GPU 1: layers 8–15; GPU 2: layers 16–23; etc. — sequential partition |
Simple Analogy
A car assembly line where each station (GPU) adds a different component: while station 2 welds the body of car #2, station 1 is already painting car #3 and station 3 is fitting the engine to car #1. All stations work simultaneously on different cars — idle time only at the start and end of the shift.
Common Usage Examples
torch.distributed.pipeline.sync.Pipe(model, chunks=8)— PyTorch pipeline parallel primitive- Megatron-LM:
--pipeline-model-parallel-size 4— split model across 4 pipeline stages - DeepSpeed:
"pipeline": {"stages": 4, "partition_method": "parameters"}— pipeline config - GPT-4 training: 3D parallelism = 8-way data × 8-way tensor × 4-way pipeline across 256 GPUs per node
- PipeDream: Microsoft Research's original pipeline parallelism paper — introduced 1F1B scheduling
Summary
In short: Pipeline parallelism assigns different model layers to different GPUs that process micro-batches like an assembly line — enabling training of models too large for any single node, at the cost of pipeline bubble idle time.