← AI Terminology
AWQ / GPTQ
AWQ (Activation-aware Weight Quantization) and GPTQ (Generative Pre-trained Transformer Quantization) are two leading post-training quantization methods that compress LLM weights to 4-bit or 3-bit precision with minimal accuracy loss.
Both enable large models to run on consumer GPUs that could not otherwise fit them.
Both enable large models to run on consumer GPUs that could not otherwise fit them.
Why It Matters in AI
Quantizing a 70B-parameter model from FP16 (2 bytes/param = 140 GB) to 4-bit (0.5 bytes/param = 35 GB) makes it fit on a single consumer GPU like an RTX 4090. AWQ and GPTQ are the two methods that achieve this compression with the least perplexity degradation, making frontier-scale models accessible for local inference and reducing cloud inference costs significantly.
Key Points
| Aspect | Description |
|---|---|
| AWQ | Identifies and protects the 1% of "salient" weights most important to accuracy; quantises the rest |
| GPTQ | Minimises quantisation error using second-order Hessian information; applied layer by layer |
| Speed | AWQ: fast inference via fused CUDA kernels; GPTQ: similar via ExLlamaV2 or AutoGPTQ kernels |
| Tools | auto-awq, AutoGPTQ, llama.cpp (GGUF), HuggingFace bitsandbytes (NF4/int8) |
| Bit depths | Typically 4-bit (W4A16 — 4-bit weights, 16-bit activations); 3-bit also possible with quality loss |
| vs bitsandbytes | AWQ/GPTQ are static (quantised once offline); bitsandbytes (QLoRA) quantises dynamically at load |
Simple Analogy
AWQ and GPTQ are like JPEG compression for model weights: they figure out which information can be discarded with the least visible quality loss and represent the rest with fewer bits. AWQ first identifies the pixels that matter most and protects them; GPTQ minimises the total squared error after compression.
Common Usage Examples
from awq import AutoAWQForCausalLM; model.quantize(tokenizer, quant_config={"w_bit": 4})- HuggingFace Hub: thousands of models available as
*-AWQor*-GPTQvariants (e.g.Llama-3-70B-Instruct-AWQ) python -m auto_gptq.modeling._base --model_name_or_path meta-llama/Meta-Llama-3-8B- llama.cpp uses GGUF format (compatible with AWQ/GPTQ-derived quantisation) for CPU+GPU inference
- Running Qwen2-72B-Instruct-AWQ on a single A100 80 GB GPU instead of requiring 4× H100
Summary
In short: AWQ and GPTQ compress LLMs to 4 bits with minimal quality loss — the key techniques that bring 70B-parameter models onto a single GPU.