← AI Terminology
LoRA - Low-Rank Adaptation
LoRA (Low-Rank Adaptation) is a parameter-efficient fine-tuning method that freezes a pre-trained model's weights and injects trainable low-rank matrix decompositions into each transformer layer — enabling fine-tuning with 10–10,000× fewer trainable parameters than full fine-tuning.
It is the dominant technique for adapting large language models to custom tasks without full fine-tuning costs.
It is the dominant technique for adapting large language models to custom tasks without full fine-tuning costs.
Why It Matters in AI
Fine-tuning a 70B-parameter model requires 140GB+ of GPU memory for weights alone — impossible on consumer hardware. LoRA inserts two small matrices (rank r ≪ d) whose product approximates the full weight update, training only those. A 7B model can be LoRA-fine-tuned on a single A100 in hours rather than days. This democratised LLM customisation: Alpaca, Vicuna, and thousands of HuggingFace LoRA adapters were created by researchers with single-GPU setups.
Key Points
| Aspect | Description |
|---|---|
| QLoRA | LoRA on a 4-bit quantised base model — fine-tune 65B models on a single 48GB GPU |
| Rank r | Typical r = 8–64; higher r = more capacity but more parameters; r=8 often sufficient |
| Adapters | Each fine-tuning task gets its own LoRA adapter — swap adapters to switch tasks on one base model |
| α (alpha) | Scaling factor: output = W₀x + (α/r) × BAx — controls contribution of LoRA matrices |
| Decomposition | Weight update ΔW ≈ A × B, where A ∈ ℝ^{d×r}, B ∈ ℝ^{r×k}, rank r ≪ min(d,k) |
| Merge at deploy | LoRA matrices can be merged into base weights at inference: W = W₀ + BA — zero latency cost |
Simple Analogy
A photo filter applied to a frozen master photograph: the original photo (base model) never changes; you add a thin overlay (LoRA matrices) that modifies how it looks for a specific style. Swap the filter for a different look — the master photo stays untouched.
Common Usage Examples
from peft import LoraConfig, get_peft_model; config = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj","v_proj"])model = get_peft_model(base_model, config)— wraps model with LoRA adapterstrainer = SFTTrainer(model, train_dataset=dataset, peft_config=lora_config)- Merge:
merged_model = model.merge_and_unload()— fuses adapters into base weights - HuggingFace Hub: thousands of LoRA adapters published per base model (e.g.
mistral-7b-instruct-lora-finance)
Summary
In short: LoRA enables fine-tuning large language models by training only a tiny fraction of parameters — making LLM customisation accessible on consumer hardware and reducing fine-tuning cost by orders of magnitude.