← AI Terminology
Full Fine-Tuning
Full fine-tuning updates all (or nearly all) model parameters on downstream data, unlike PEFT methods that freeze most weights.
It maximises capacity to change behaviour at highest compute/storage cost.
It maximises capacity to change behaviour at highest compute/storage cost.
Why It Matters in AI
When tasks demand deep behavioural change, full FT still wins. It needs more GPUs, risks forgetting, and produces large checkpoints — the baseline PEFT methods try to approximate.
Key Points
| Aspect | Description |
|---|---|
| Alt | LoRA/QLoRA/adapters for efficiency |
| Cost | Max memory, long trains, large artifacts |
| Risk | Catastrophic forgetting; overfitting small sets |
| Quality | Often best if data/compute allow |
| Updates | Every weight trainable |
| Practice | Common for mid-size models; rare for 70B+ without sharding |
Simple Analogy
Renovating an entire house rather than adding a modular kitchen unit — maximum change, maximum cost and disruption.
Common Usage Examples
- FSDP full FT of 7B on domain data
- Compare to LoRA on same dataset
- Save full checkpoint shards
- Mix replay data to limit forgetting
Summary
In short: Full fine-tuning updates all model weights — maximum adaptability at maximum compute and storage cost.