← AI Terminology
Data Deduplication
Data deduplication removes exact or near-duplicate documents/examples from training or evaluation sets to improve efficiency and reduce memorisation/contamination.
Standard stage in modern LLM data pipelines.
Standard stage in modern LLM data pipelines.
Why It Matters in AI
Web crawls repeat the same pages endlessly. Dupes waste compute and inflate eval scores via memorisation. MinHash/LSH and exact hashing are mandatory plumbing for serious pretraining.
Key Points
| Aspect | Description |
|---|---|
| Eval | Also dedupe train vs test contamination |
| Exact | Hash full docs or normalised text |
| Tools | datasketch, custom Spark/Beam jobs |
| Related | MinHash, Common Crawl filtering |
| Benefits | Less memorisation; better unique signal per FLOP |
| Near-dupe | MinHash, SimHash, embedding clusters |
Simple Analogy
Weeding identical flyers from a mailbox before studying the news — stop rereading the same ad 500 times.
Common Usage Examples
- MinHash LSH near-dupe removal at scale
- Exact URL/content hash drops
- Check eval leakage against train
- Report % removed in data cards
Summary
In short: Data deduplication removes exact and near-duplicate text — saving compute and reducing memorisation in LLM training.