← AI Terminology
PEFT - Parameter-Efficient Fine-Tuning
PEFT (Parameter-Efficient Fine-Tuning) refers to a family of techniques that adapt a large pre-trained model to new tasks by training only a small subset of parameters — rather than updating all weights — dramatically reducing the compute, memory, and storage costs of fine-tuning.
LoRA, QLoRA, prompt tuning, prefix tuning, and adapters are all PEFT methods.
LoRA, QLoRA, prompt tuning, prefix tuning, and adapters are all PEFT methods.
Why It Matters in AI
Full fine-tuning a 70B model requires 70B parameters × 4 bytes × (model + gradients + optimizer states) ≈ 840GB+ of GPU memory — requiring dozens of A100s for weeks. PEFT methods train 0.1–1% of parameters, reducing memory requirements by 10–100× and enabling fine-tuning on a single GPU. PEFT also enables storing many task-specific adaptations (e.g. 100 domain adapters) as small delta files on top of one base model, rather than 100 full model copies.
Key Points
| Aspect | Description |
|---|---|
| LoRA | Low-rank matrix decomposition of weight updates — most popular PEFT method; r=8–64 |
| QLoRA | LoRA on a 4-bit quantised base model — fine-tune 65B on a single GPU |
| Prefix tuning | Learn task-specific prefix key/value pairs per layer — more expressive than prompt tuning |
| Prompt tuning | Learn a small set of soft token embeddings prepended to the input — base model frozen |
| Adapter layers | Insert small trainable MLP modules between frozen transformer layers |
| HuggingFace PEFT | pip install peft — unified library for LoRA, IA3, AdaLoRA, prompt tuning, prefix tuning |
Simple Analogy
Customising a Swiss watch by replacing only the hands and dial (PEFT) vs. rebuilding the entire movement from scratch (full fine-tuning). The core mechanism (base model) stays untouched — you make targeted adjustments that personalise the watch for a specific owner (task) at a fraction of the cost.
Common Usage Examples
from peft import get_peft_model, LoraConfigconfig = LoraConfig(r=16, lora_alpha=32, target_modules=["q_proj", "v_proj"], lora_dropout=0.05)peft_model = get_peft_model(base_model, config); peft_model.print_trainable_parameters()- QLoRA:
BitsAndBytesConfig(load_in_4bit=True)+ LoraConfig — fine-tune 70B on 2× 24GB GPU peft_model.save_pretrained("./lora-adapter/")— save only the small adapter weights (~20MB for 7B model)
Summary
In short: PEFT trains only a tiny fraction of a model's parameters to adapt it to new tasks — enabling LLM fine-tuning on consumer hardware, with LoRA as the dominant method and HuggingFace PEFT as the standard library.