← AI Terminology
Model Soup
Model soup averages the weights of multiple fine-tuned models (often from different hyperparameters) that share an initialisation, often improving accuracy without extra inference cost.
Introduced by Wortsman et al. for vision and adopted more broadly.
Introduced by Wortsman et al. for vision and adopted more broadly.
Why It Matters in AI
Instead of picking the single best run, soup ingredients that each did well. Averaging in weight space can outperform the best individual fine-tune — free ensemble-like gains at deployment.
Key Points
| Aspect | Description |
|---|---|
| Pro | One model at inference; often better OOD |
| Need | Same architecture and compatible training |
| Paper | Model soups (Wortsman et al., 2022) |
| Recipe | Finetune many; average weights of selected runs |
| Related | Model merging, stochastic weight averaging |
| Uniform soup | Average all; greedy soup selects subset |
Simple Analogy
Taste-testing many good chili batches and serving a blend of the best pots — one bowl, shared strengths.
Common Usage Examples
- Average top-5 finetune checkpoints
- Greedy soup: add models that help held-out acc
- Deploy single souped weights
- Compare to output-ensemble cost
Summary
In short: Model soup averages fine-tuned weights into one stronger model — ensemble benefits without ensemble inference cost.