← AI Terminology
Epoch
An epoch is one complete pass through the entire training dataset — every training example has been seen exactly once by the model.
It is the primary unit for measuring training duration alongside steps/iterations.
It is the primary unit for measuring training duration alongside steps/iterations.
Why It Matters in AI
Epochs define the training schedule and are the natural unit for monitoring training progress. Too few epochs and the model undertrains (high bias); too many and it overfits (high variance, especially on small datasets). In practice, the right number of epochs varies enormously: image classifiers train for 100–300 epochs; LLM pre-training runs through the dataset 1–3 times across trillions of tokens.
Key Points
| Aspect | Description |
|---|---|
| Validation | Evaluate on validation set every N epochs or steps to monitor training progress |
| Relationship | 1 epoch = (dataset size / batch size) gradient update steps |
| Epoch vs step | LLM training often tracked in steps (gradient updates) or tokens seen, not epochs |
| Early stopping | Halt after patience epochs of no validation improvement — prevents over-epoching |
| Large datasets | LLM pre-training: often < 1 full epoch — dataset is so large one pass suffices |
| Small datasets | May need 100s of epochs before convergence — each example seen many times |
Simple Analogy
Reading a textbook: one epoch is reading it cover to cover once. Training for 10 epochs is reading the same book 10 times — each time you catch nuances you missed. For a very long book (large dataset), even one reading may take months; for a short book (small dataset), you need many re-reads to learn it thoroughly.
Common Usage Examples
model.fit(X_train, y_train, epochs=50, validation_split=0.2)in Kerasfor epoch in range(num_epochs): train_one_epoch(model, loader)— PyTorch training loop--num_train_epochs 3in HuggingFace Trainer for fine-tuning LLMs (3 epochs typical for instruction tuning)- LLM pre-training: Llama 3 trained on ~15T tokens — roughly 1–2 epochs over the training corpus
- Cosine annealing: LR schedule decays to 0 over the total number of epochs — epoch count defines the schedule
Summary
In short: An epoch is one full pass through the training data — the measuring stick for how long a model has trained and when to stop.