← AI Terminology
Quantization-Aware Training
Quantization-aware training (QAT) is a training technique that simulates low-precision quantization during the forward pass — exposing the model to quantization noise during training so it learns to be robust to the precision reduction before deployment.
It typically produces higher-accuracy quantized models than post-training quantization (PTQ), at the cost of a training run.
It typically produces higher-accuracy quantized models than post-training quantization (PTQ), at the cost of a training run.
Why It Matters in AI
Post-training quantization (applying quantization after training) is fast but causes accuracy drops, especially at INT4 or lower. QAT re-trains or fine-tunes the model with fake quantization layers that round weights during the forward pass but maintain full precision for gradient updates — teaching the model to find weight configurations that are robust to being rounded. MobileNet, EfficientNet edge deployments and many on-device models use QAT to achieve INT8 accuracy within 1% of FP32.
Key Points
| Aspect | Description |
|---|---|
| vs PTQ | QAT: requires training, better accuracy; PTQ: no training, faster to deploy, more degradation |
| LLM QAT | Computationally expensive for 70B+ models — LLM-QAT (2023) showed it recovers W4A8 performance |
| Deployment | QAT model exported to ONNX or TFLite and deployed with INT8 fixed-point kernels |
| Calibration | PTQ requires calibration data for scale/zero-point estimation; QAT learns them during training |
| Straight-through | Gradient passes through the rounding operation as if it were the identity — enables backprop |
| Fake quantization | Forward pass: round weights to INT8/INT4 grid; backward pass: straight-through estimator (full precision) |
Simple Analogy
Rehearsing a speech with noise-cancelling headphones off: if you always practice in silence (FP32 training) then deliver in a noisy room (quantized deployment), you'll stumble. QAT is practicing with the noise present — the speaker adapts and learns to be understood despite the interference.
Common Usage Examples
- PyTorch QAT:
model.qconfig = torch.quantization.get_default_qat_qconfig('fbgemm') torch.quantization.prepare_qat(model, inplace=True)— insert fake quant nodes- Train for a few epochs, then:
torch.quantization.convert(model, inplace=True)— deploy INT8 model - TensorFlow:
tfmot.quantization.keras.quantize_model(model)— TF Model Optimization Toolkit QAT - MobileNetV2 INT8 QAT: within 0.5% top-1 accuracy of FP32 baseline on ImageNet
Summary
In short: Quantization-aware training simulates quantization noise during training so the model learns to tolerate precision reduction — producing significantly higher-accuracy quantized models than post-training quantization, at the cost of an additional training run.