← AI Terminology
Prefix Tuning
Prefix tuning learns continuous task-specific vectors prepended to keys/values (or embeddings) while freezing the base model — optimising “soft prompts” in activation space.
A PEFT method from Li & Liang (2021).
A PEFT method from Li & Liang (2021).
Why It Matters in AI
Discrete prompt engineering is brittle; learned prefixes adapt deep layers with few parameters. It influenced prompt tuning and modern soft-control methods for frozen LLMs.
Key Points
| Aspect | Description |
|---|---|
| Use | Task packs on shared frozen LLM |
| Idea | Trainable prefix tokens influencing attention |
| Scope | Often per layer K/V prefixes |
| Params | Very small vs full finetune |
| Related | Prompt tuning, P-Tuning, PEFT |
| Vs LoRA | Different locus (activations vs weight deltas) |
Simple Analogy
Handing a frozen expert a short learned briefing packet before each type of case — the expert’s brain unchanged, the briefing steers behaviour.
Common Usage Examples
- PEFT
PrefixTuningConfig - Train prefixes on summarisation only
- Store small prefix files per task
- Compare to LoRA on same budget
Summary
In short: Prefix tuning learns soft continuous prefixes that steer a frozen model — PEFT via activations rather than full weight updates.