← AI Terminology
Batch
A batch is a subset of the training dataset processed together in one forward and backward pass before updating the model's weights.
Batch size is one of the most consequential hyperparameters in training: it affects memory use, convergence speed, and final model quality.
Batch size is one of the most consequential hyperparameters in training: it affects memory use, convergence speed, and final model quality.
Why It Matters in AI
Training on the full dataset at once (batch gradient descent) is memory-prohibitive and wasteful; training on one sample at a time (online learning) is noisy and slow. Batches strike the balance: enough samples for a stable gradient estimate, few enough to fit in GPU memory and iterate quickly. Batch size also has a regularisation effect — smaller batches add noise that can help escape sharp minima.
Key Points
| Aspect | Description |
|---|---|
| GPU memory | Batch size limited by VRAM — larger batches require gradient accumulation or multi-GPU setups |
| Mini-batch | Typical: 32–512 samples — standard in practice; balances stability and speed |
| Global batch | In distributed training, global batch = per-device batch × number of devices |
| Full batch GD | All training data — stable gradients, huge memory, slow updates, can converge to sharp minima |
| Learning rate | Should scale with batch size (linear scaling rule) — larger batches often need larger LR |
| Stochastic (SGD) | Batch size = 1 — maximum noise, fastest updates, but high variance |
Simple Analogy
Grading exams: batch gradient descent reads all 1,000 papers before adjusting the marking rubric; online learning adjusts after every single paper; mini-batch reads 32 at a time — quick feedback without overreacting to any one unusual paper.
Common Usage Examples
DataLoader(dataset, batch_size=64, shuffle=True)in PyTorch- Gradient accumulation:
loss = loss / accumulation_steps; loss.backward()— simulate larger batches with less VRAM - LLM training: global batch sizes of 1–4M tokens are typical for pre-training runs
--per_device_train_batch_size 8 --gradient_accumulation_steps 4in HuggingFace Trainer- Batch size finder:
trainer.tune(model)in PyTorch Lightning auto-selects largest fitting batch size
Summary
In short: A batch is the chunk of data processed together before each weight update — choosing the right size balances memory, speed, and convergence quality.