← AI Terminology
Data Mixture
Data mixture is the proportional recipe of sources/domains in a training run (web, code, math, multilingual, books) and how those weights change over training.
It is among the highest-leverage LLM training decisions.
It is among the highest-leverage LLM training decisions.
Why It Matters in AI
Architecture papers get fame; mixture often decides skills. Upsampling code/math mid-training or balancing languages changes capability profiles. Open reports increasingly publish mixture schedules.
Key Points
| Aspect | Description |
|---|---|
| Axes | Domain weights, quality tiers, languages, stages |
| Risk | Overfit mixture to public benches |
| Effects | Coding skill, toxicity, multilinguality, refusal |
| Related | Mid-training, FineWeb, scaling laws with data |
| Practice | Ablate mixes on small proxies first |
| Schedules | Constant mix vs curriculum/annealing |
Simple Analogy
A chef’s ratio of ingredients in a broth — more chili vs more stock completely changes the dish even with the same cooking pot (model).
Common Usage Examples
- Publish % code/math/web in model cards
- Upsample domain data in mid-training
- Proxy small-model mixture sweeps
- Watch safety metrics when upsampling web
Summary
In short: Data mixture is the weighted recipe of training sources — often as decisive for model skills as architecture itself.