← AI Terminology
BF16 - Brain Floating Point
BF16 (bfloat16) is a 16-bit float with the same exponent range as FP32 but fewer mantissa bits — popularised for deep learning stability on TPUs and GPUs.
It is a default mixed-precision format for LLM training.
It is a default mixed-precision format for LLM training.
Why It Matters in AI
FP16 overflows easily in large models; BF16’s range stabilises training with minimal code change. Most modern LLM recipes train in BF16 (or FP8) automatically.
Key Points
| Aspect | Description |
|---|---|
| Serve | Also common inference dtype before int quant |
| Layout | 1 sign, 8 exp, 7 mantissa |
| PyTorch | torch.bfloat16, autocast |
| Related | Mixed precision, FP8, Tensor Cores |
| Vs FP16 | Wider range, less precision |
| Origin use | Google Brain / TPU ecosystem; now NVIDIA too |
Simple Analogy
A measuring cup marked for huge dynamic range but coarser fine lines — less likely to overflow, slightly less precise sips.
Common Usage Examples
model.to(torch.bfloat16)- Autocast BF16 training loops
- Check GPU BF16 support
- LLM configs:
torch_dtype=bfloat16
Summary
In short: BF16 is the brain-float 16-bit format that stabilises mixed-precision deep learning with FP32-like range.