← AI Terminology
Adapter Layers
Adapter layers are small trainable modules inserted into a frozen pretrained network (often bottleneck MLPs after attention/FFN) so task adaptation needs few new parameters.
They pioneered parameter-efficient transfer before LoRA’s popularity.
They pioneered parameter-efficient transfer before LoRA’s popularity.
Why It Matters in AI
Full fine-tuning is heavy; adapters specialise models cheaply and can be swapped per task. They established the PEFT paradigm still used in multi-tenant serving (task-specific modules on a shared base).
Key Points
| Aspect | Description |
|---|---|
| Con | Adds small depth/latency vs pure LoRA |
| Pro | Tiny trainable params; modular tasks |
| Use | Multi-task NLP, domain packs |
| Origin | Houlsby adapters (2019) and variants |
| Related | LoRA, prefix tuning, PEFT libraries |
| Structure | Down-project → nonlinearity → up-project residual |
Simple Analogy
Clipping a small specialised lens onto a fixed camera body for each job — the big camera stays shared, the lens swaps.
Common Usage Examples
- Insert Houlsby adapters in each transformer block
- Train only adapter weights on a task
- Swap adapters at inference per tenant
- Compare param % vs full finetune
Summary
In short: Adapter layers are tiny modules added to a frozen model for cheap task specialisation — early PEFT that still shapes multi-task serving.