← AI Terminology
Iteration
Iteration in machine learning refers to one pass of the optimiser over a single mini-batch of data — updating model weights once using the gradients computed from that batch.
One epoch = multiple iterations; iterations per epoch = dataset size ÷ batch size.
One epoch = multiple iterations; iterations per epoch = dataset size ÷ batch size.
Why It Matters in AI
Tracking iterations (rather than just epochs) matters for learning rate scheduling, gradient accumulation, and logging frequency — especially when datasets are huge and a single epoch takes hours. Modern frameworks express schedules in steps (iterations): "warmup for 1000 steps, then cosine decay to 0 over 100,000 steps." Understanding iteration vs. epoch vs. step prevents misconfigured training runs.
Key Points
| Aspect | Description |
|---|---|
| Logging | Typically log loss/metrics every N iterations rather than every epoch for large datasets |
| Convergence | Training converges over thousands to millions of iterations depending on task scale |
| LR scheduling | Most modern schedulers (cosine, linear warmup) operate per-step, not per-epoch |
| Iteration = step | One forward + backward pass + weight update on one batch — same as "training step" |
| Epoch relationship | Epochs = iterations × batch_size / dataset_size (rearranged to suit) |
| Gradient accumulation | Splits one effective batch into K micro-batches; weight update happens every K iterations |
Simple Analogy
A student working through a practice exam one question at a time: each question is an iteration. Finishing the whole exam is one epoch. The student improves slightly after each question (weight update), not just at the end of the exam.
Common Usage Examples
- PyTorch training loop:
for batch in dataloader: optimizer.zero_grad(); loss.backward(); optimizer.step()— each loop = one iteration scheduler = get_cosine_schedule_with_warmup(optimizer, num_warmup_steps=1000, num_training_steps=100000)- Hugging Face
Trainer(max_steps=50000)— total iteration count, not epochs global_stepcounter in TensorBoard: x-axis of training curves in step (iteration) units- Gradient accumulation:
if step % accumulation_steps == 0: optimizer.step(); optimizer.zero_grad()
Summary
In short: An iteration is one weight update — the atomic unit of training — and most modern learning rate schedules, logging, and convergence analysis operate in terms of iterations rather than epochs.