← AI Terminology
Model Collapse
Model collapse is a failure mode in which AI models trained on AI-generated data progressively lose diversity and accuracy — because each generation of model reinforces and amplifies the biases and errors of the previous one, eventually producing degenerate outputs that bear little resemblance to the original data distribution.
As AI-generated content floods the internet, model collapse is an emerging existential risk for future pre-training.
As AI-generated content floods the internet, model collapse is an emerging existential risk for future pre-training.
Why It Matters in AI
The internet is being flooded with AI-generated text and images. If future models are trained on this data without filtering, they inherit not just the knowledge of their predecessors but also their hallucinations, biases, and lack of diversity. Each training generation amplifies these errors until outputs become repetitive and wrong. Research (Shumailov et al. 2023) demonstrated this mathematically — model collapse is a real failure mode, not a hypothetical, driving interest in data provenance and synthetic data auditing.
Key Points
| Aspect | Description |
|---|---|
| Provenance | Data watermarking and provenance tracking are proposed defences — know what's AI-generated |
| Core erosion | Late collapse: even common patterns degrade as errors compound across generations |
| Tail erosion | Early collapse: rare but real data patterns (long-tail) are lost — model forgets edge cases |
| Partial mitigation | Mixing human and synthetic data carefully; filtering for quality; diversity-preserving sampling |
| Human data scarcity | Human-generated text is now a scarce premium resource — a competitive moat for labs with it |
| Synthetic data risk | Training on unfiltered AI-generated data accelerates collapse — requires careful curation |
Simple Analogy
A game of telephone where each person only remembers the gist: the original message ("the cat sat on the mat at noon") becomes "a cat was somewhere" by generation 5 and "animals exist" by generation 10. Each retelling loses detail, amplifies common patterns, and eliminates nuance until the original information is unrecoverable.
Common Usage Examples
- Shumailov et al. 2023: "The Curse of Recursion" — first formal proof of model collapse in generative models
- Common Crawl quality: scrapers now apply AI-content filters to preserve human-text quality
C4dataset curation: removes near-duplicate and low-quality content to prevent early collapse signals- Synthetic data best practice: augment with AI-generated data only for rare/underrepresented cases
- OpenAI, Anthropic internal policies: audit training data for AI-generated content fraction
Summary
In short: Model collapse is the progressive degradation of AI models trained on AI-generated data — each generation amplifying errors until outputs lose diversity and accuracy, making human data provenance critically important.