← AI Terminology
Common Crawl
Common Crawl is a nonprofit’s massive ongoing scrape of the public web, published as open WARC datasets widely used to train language models.
It is a primary raw source behind many LLM pretraining corpora.
It is a primary raw source behind many LLM pretraining corpora.
Why It Matters in AI
Frontier and open LMs drink from web-scale text. Common Crawl is the public firehose — noisy, powerful, and ethically/legally debated. Understanding filtering from CC to FineWeb-like sets is data engineering for LLMs.
Key Points
| Aspect | Description |
|---|---|
| Ops | Petabyte-scale processing |
| Use | Pretraining mixtures after heavy filtering |
| What | Periodic web crawls, free to download |
| Issues | Noise, PII, toxicity, copyright questions |
| Related | The Pile, RefinedWeb, FineWeb, C4 |
| Pipeline | Dedup, language ID, quality filters |
Simple Analogy
A public warehouse of vacuumed web pages continuously refilled — raw material factories refine into training fuel.
Common Usage Examples
- Download CC WARC segments
- Language filter + quality score pipelines
- Build domain subsets from CC
- Cite CC in dataset cards
Summary
In short: Common Crawl is the open web-scale crawl feeding most LLM pretraining pipelines after aggressive filtering.