← AI Terminology
DoRA - Weight-Decomposed Low-Rank Adaptation
DoRA decomposes weights into magnitude and direction and applies LoRA-style updates primarily to direction, improving expressivity over vanilla LoRA in many finetunes.
Proposed as a drop-in upgrade path for LoRA-style PEFT.
Proposed as a drop-in upgrade path for LoRA-style PEFT.
Why It Matters in AI
LoRA is dominant but not always optimal. DoRA’s magnitude/direction split often yields better accuracy for similar rank budgets — relevant when squeezing quality from small adapter ranks.
Key Points
| Aspect | Description |
|---|---|
| Pro | Often beats LoRA at same rank |
| Cost | Slightly more compute/memory than LoRA |
| Idea | Weight = magnitude × direction; adapt with low-rank on direction |
| Origin | Liu et al., DoRA (2024) |
| Related | LoRA, QLoRA, adapters |
| Ecosystem | Supported in PEFT libraries |
Simple Analogy
Tuning a speaker by adjusting how loud it is separately from the shape of the sound wave — two knobs instead of one muddled control.
Common Usage Examples
- PEFT
LoraConfigwith DoRA flag where available - Compare LoRA rank 8 vs DoRA rank 8
- Merge DoRA adapters for deploy
- Use on LLM instruction finetunes
Summary
In short: DoRA is a LoRA variant that splits magnitude and direction — often stronger fine-tunes at the same low rank.