← AI Terminology

Quantization

Quantization is a model compression technique that represents model weights and/or activations in lower-precision numeric formats (INT8, INT4, FP8) instead of FP32 or FP16 — reducing model memory footprint and enabling faster inference on hardware with integer arithmetic units.

It is the most practical and widely deployed inference optimisation technique.
Why It Matters in AI
A Llama 3 70B model in FP16 requires ~140GB of GPU memory — exceeding a single A100. In INT4 (4-bit), the same model fits in ~35GB — a single A100 80GB. Quantization also speeds up inference: INT8 matrix multiplications run 2–4× faster than FP16 on GPU tensor cores and CPU SIMD units. Consumer-grade local AI (Ollama, llama.cpp) is entirely dependent on quantization — without it, running 7B+ models on laptops would be impossible.
Key Points
Aspect Description
AWQ Activation-aware Weight Quantisation: scale weights by activation magnitude — better accuracy
GGUF File format for quantised LLMs in llama.cpp — Q4_K_M, Q8_0 etc. indicate quantisation level
GPTQ Post-training quantisation for LLMs: layer-by-layer reconstruction minimising quantisation error
Weight-only Quantise weights to INT4/INT8; dequantise to FP16 for computation — simplest approach
Activation quant Quantise activations during inference — harder (outliers), more hardware speedup
Quality tradeoff Q8: ~0% perplexity increase; Q4: ~1–3%; Q2: significant degradation — sweet spot is Q4–Q5
Simple Analogy
MP3 audio vs. WAV: WAV stores every sample at full precision (FP32); MP3 compresses by discarding inaudible detail (quantisation). A well-tuned Q4 model is like a high-bitrate MP3 — almost indistinguishable from the original in practice, but dramatically smaller.
Common Usage Examples
  • llama.cpp: ./llama-cli -m llama-3-8b.Q4_K_M.gguf -p "Hello" — 4-bit quantised local inference
  • GPTQ: from auto_gptq import AutoGPTQForCausalLM; model = AutoGPTQForCausalLM.from_quantized(...)
  • AWQ: from awq import AutoAWQForCausalLM; model = AutoAWQForCausalLM.from_quantized("llama-3-8b-awq")
  • BitsAndBytes: BitsAndBytesConfig(load_in_8bit=True) — 8-bit inference in HuggingFace
  • Ollama: automatically downloads and runs quantised GGUF models — ollama run llama3 uses Q4_K_M
Summary
In short: Quantization represents model weights in lower-precision formats (INT4/INT8) to dramatically reduce memory and accelerate inference — the essential technique that makes running large LLMs on consumer hardware possible.