← AI Terminology
Continued Pretraining
Continued pretraining (domain-adaptive pretraining) further trains a foundation model with the original self-supervised objective on new raw corpora before task fine-tuning.
It injects domain language without full training from scratch.
It injects domain language without full training from scratch.
Why It Matters in AI
When the domain is large and unlabeled (legal PDFs, code monorepos), continued next-token training beats jumping straight to small SFT sets. It is a standard intermediate stage in specialised LLM pipelines.
Key Points
| Aspect | Description |
|---|---|
| Cost | More compute than SFT; far less than training from scratch |
| Data | Large unlabeled domain corpus |
| Then | SFT/RLHF for behaviour |
| Related | Mid-training, domain adaptation |
| Objective | Usually next-token or masked LM still |
| Vs SFT only | Better domain fluency and terminology |
Simple Analogy
An experienced generalist doctor doing an extra residency year reading only cardiology literature before practicing as a cardiologist.
Common Usage Examples
- Pretrain longer on code before coding SFT
- Legal corpus continued pretrain → counsel assistant SFT
- Watch general benchmarks for regression
- Mix a % general data to reduce forgetting
Summary
In short: Continued pretraining extends foundation training on domain text — specialising language skill before instruction tuning.