← AI Terminology
Scaling Laws
Scaling laws are empirical relationships showing that the performance of language models improves predictably as a power law with increases in model parameters (N), training data tokens (D), and compute budget (C) — enabling researchers to predict model quality before training, and driving the strategy of continuously scaling up AI.
The Chinchilla scaling law is the current standard for optimal compute allocation.
The Chinchilla scaling law is the current standard for optimal compute allocation.
Why It Matters in AI
Scaling laws transformed AI from an experimental art to a (partially) predictable engineering discipline: given a compute budget, the laws tell you the optimal model size and dataset size to train. They explain why frontier labs spend billions on training runs rather than architecture innovation — more compute predictably improves capability. The discovery that emergent abilities appear at certain scales drives competition to reach those thresholds, shaping the entire AI investment landscape.
Key Points
| Aspect | Description |
|---|---|
| Data wall | High-quality text corpora are finite — scaling laws assume more data is available; synthetic data is a proposed solution |
| Beyond LLMs | Scaling laws observed in code models, vision models, and multimodal models — appear universal |
| Chinchilla rule | For C FLOPs, optimal: N ∝ √C parameters, D ∝ √C tokens — most models before 2022 were undertrained |
| Chinchilla (2022) | Hoffman et al. (DeepMind): optimal compute split is roughly equal between model size and tokens |
| Emergent abilities | Capabilities (arithmetic, CoT, 5-shot MMLU) appear suddenly at certain parameter/compute scales |
| Kaplan et al. 2020 | Original OpenAI scaling laws — loss scales as power law with N, D, C: L ∝ N^{-α} |
Simple Analogy
Moore's Law for AI quality: just as transistor density predicted computer performance for decades, scaling laws predict model quality from compute budget. You don't need to discover a new architecture — spend more compute, get a better model, reliably and predictably.
Common Usage Examples
- Chinchilla optimal: LLaMA-3 70B was trained on ~15T tokens (over-trained per Chinchilla — for inference efficiency)
loss = C₀ * N^{-0.076}— approximate scaling with model parameters (Kaplan form)compute_optimal_size(budget_flops)— estimate optimal N and D from total compute C- Frontier labs use scaling law extrapolation to justify multi-billion dollar training investments
- Emergent abilities chart (Wei et al., 2022): BIG-Bench tasks showing sudden capability appearance at scale
Summary
In short: Scaling laws are the empirical power-law relationships between compute, model size, data, and AI quality — enabling predictable capability improvement through resource scaling and driving the frontier AI industry's strategy of continuous scale increases.