← AI Terminology
Mixed Precision Training
Mixed precision training is a technique that uses lower-precision floating-point formats (FP16 or BF16) for most computations while maintaining FP32 "master weights" for gradient accumulation and updates — halving memory usage and 2–4× accelerating matrix multiplications on modern tensor core hardware.
It is standard practice for all large-scale deep learning training today.
It is standard practice for all large-scale deep learning training today.
Why It Matters in AI
FP32 training of a 70B-parameter model requires ~280GB of memory — far beyond a single GPU. FP16/BF16 reduces this by half, and Tensor Cores on NVIDIA A100/H100 execute FP16/BF16 matrix multiplications 2–8× faster than FP32. Without mixed precision, large model training would be infeasible at its current scale. It is enabled by default in HuggingFace Trainer, PyTorch Lightning, and every major LLM training framework.
Key Points
| Aspect | Description |
|---|---|
| BF16 | 8-bit exponent, 7-bit mantissa — same range as FP32, less prone to overflow; preferred for LLMs |
| FP16 | 5-bit exponent, 10-bit mantissa — can overflow for large values; requires loss scaling |
| Loss scaling | Scale loss up before backward pass (FP16) to prevent gradient underflow — scaled back after |
| Master weights | FP32 copy of weights kept for gradient updates — ensures accumulation doesn't lose precision |
| Memory savings | ~2× model memory reduction; enables 2× larger models or batch sizes per GPU |
| Hardware support | NVIDIA Ampere (A100), Hopper (H100), Ada; AMD MI300X; Google TPUv4/v5 — all have BF16 acceleration |
Simple Analogy
Writing a novel: draft chapters in shorthand (FP16/BF16 — fast, compact) but keep a master copy in full prose (FP32) for final editing and consistency. The shorthand drafts are good enough for fast iteration; the master copy preserves precision where it counts.
Common Usage Examples
scaler = torch.cuda.amp.GradScaler(); with torch.autocast("cuda"): loss = model(batch)— PyTorch AMP- HuggingFace:
TrainingArguments(bf16=True)— BF16 training on Ampere+ GPUs - DeepSpeed:
"fp16": {"enabled": true, "loss_scale": 0}— auto loss scaling torch.set_float32_matmul_precision("high")— enables TF32 for matmuls on Ampere- FSDP:
MixedPrecision(param_dtype=bfloat16, reduce_dtype=float32)— ZeRO + mixed precision
Summary
In short: Mixed precision training uses FP16/BF16 for fast tensor core computation while keeping FP32 master weights for stable gradient updates — halving memory and 2–4× accelerating training, making it standard practice for all large model training.