← AI Terminology
QLoRA - Quantized Low-Rank Adaptation
QLoRA is a fine-tuning method that combines 4-bit quantisation of the base model with LoRA adapters — enabling fine-tuning of models up to 65B parameters on a single consumer GPU (e.g. 48GB A100 or 24GB RTX 4090) with performance comparable to full-precision fine-tuning.
It is the key technique that made large LLM fine-tuning accessible to researchers without data centre hardware.
It is the key technique that made large LLM fine-tuning accessible to researchers without data centre hardware.
Why It Matters in AI
Before QLoRA (Dettmers et al., 2023), fine-tuning a 65B model required ~780GB of GPU memory across multiple A100s. QLoRA reduced this to ~48GB by: (1) quantising the frozen base model to 4-bit NF4 precision, (2) training only small LoRA adapters in 16-bit, (3) using paged optimisers to manage CPU memory. This democratised large LLM customisation: Guanaco 65B, trained on a single GPU in 24 hours, performed comparably to ChatGPT on the Vicuna benchmark.
Key Points
| Aspect | Description |
|---|---|
| NF4 | NormalFloat 4-bit: quantisation format designed for normally distributed weights — better than INT4 |
| Memory | 65B model in QLoRA: ~48GB; 13B: ~16GB; 7B: ~10GB — single-GPU feasible for all sizes |
| Performance | Within 1–2% of full 16-bit fine-tuning on instruction following benchmarks (Alpaca, Vicuna) |
| LoRA adapters | Trained in BF16 on top of frozen 4-bit base model — de-quantised during forward pass |
| Paged optimizers | CPU RAM used to store GPU optimizer states that exceed GPU memory — prevents OOM crashes |
| Double quantisation | Quantise the quantisation constants themselves — saves ~0.37 bits/parameter additional memory |
Simple Analogy
Renovating a house while storing the furniture in compression bags: the furniture (base model weights) takes up 4× less space when vacuum-packed (4-bit quantisation), freeing room for the renovation work (LoRA adapters trained in full precision). Unpack each piece only when you need to use it.
Common Usage Examples
from transformers import BitsAndBytesConfigquantization_config = BitsAndBytesConfig(load_in_4bit=True, bnb_4bit_quant_type="nf4", bnb_4bit_compute_dtype=torch.bfloat16)model = AutoModelForCausalLM.from_pretrained("meta-llama/Llama-3-8B", quantization_config=quantization_config)model = get_peft_model(model, LoraConfig(r=16, target_modules=["q_proj","v_proj"]))— add LoRA on toptrainer = SFTTrainer(model, train_dataset=dataset, peft_config=lora_config)— full QLoRA pipeline
Summary
In short: QLoRA combines 4-bit base model quantisation with LoRA adapters, enabling fine-tuning of 65B+ parameter models on a single GPU — the key breakthrough that democratised large LLM customisation for researchers and practitioners.