← AI Terminology
MinHash / LSH
MinHash estimates Jaccard similarity between sets via hash signatures; LSH (locality-sensitive hashing) buckets similar items together for sublinear near-duplicate search.
Together they power large-scale text deduplication.
Together they power large-scale text deduplication.
Why It Matters in AI
Comparing every document pair is impossible at web scale. MinHash+LSH finds likely duplicates efficiently — the algorithmic core of C4/Pile/FineWeb-style pipelines.
Key Points
| Aspect | Description |
|---|---|
| LSH | Hash so similar items collide in buckets |
| Use | Near-dupe detection, clustering rough matches |
| Libs | datasketch, custom distributed impls |
| Params | Num permutations, bands/rows trade recall/precision |
| MinHash | Signature for set similarity estimation |
| Related | Data deduplication, ANN for embeddings |
Simple Analogy
Fingerprinting books by a few random page samples and shelving similar fingerprints near each other to catch reprints quickly.
Common Usage Examples
datasketch.MinHash+MinHashLSH- Spark jobs for CC dedup
- Tune bands for recall targets
- Secondary exact check after LSH candidates
Summary
In short: MinHash and LSH estimate set similarity and find near-duplicates in sublinear time — essential algorithms for web-scale data cleaning.