← AI Terminology
Training
Training is the process of optimising a model's parameters by repeatedly presenting examples from the training dataset, computing a loss measuring prediction error, backpropagating gradients through the model, and updating weights to reduce that error — iterating until the model achieves satisfactory performance.
It is the process that creates model knowledge from data.
It is the process that creates model knowledge from data.
Why It Matters in AI
Training is both the most computationally expensive and most strategically important step in AI: GPT-4 training cost an estimated $100M+ in compute; Llama 3 405B took ~30M GPU hours. The quality of data, architecture, loss function, and optimiser determines what the model learns. Training is one-time: pay the cost once, deploy the weights millions of times. Understanding training — what makes it stable, when it converges, how to diagnose failures — is the core ML engineering skill.
Key Points
| Aspect | Description |
|---|---|
| Epoch | One full pass through the training dataset — typically many epochs (10–100) or steps (100K–1M) |
| Monitoring | Track train/val loss curves, learning rate schedule, gradient norms — detect instability early |
| Forward pass | Input → model → prediction → loss computation |
| Backward pass | loss.backward() — compute gradients via backpropagation through the computation graph |
| Training loop | Repeat: sample mini-batch → forward → loss → backward → update → until convergence |
| Weight update | optimizer.step() — adjust weights proportional to gradients |
Simple Analogy
Teaching someone to ride a bike: fall (loss), feel where balance was off (gradient), adjust posture (weight update), try again. Repeat thousands of times across many different terrain types (mini-batches, epochs) until balance is automatic (convergence). The knowledge of how to balance (weights) is encoded in muscle memory (model parameters).
Common Usage Examples
- PyTorch training loop:
for epoch in range(100): for batch in loader: optimizer.zero_grad(); loss = model(batch); loss.backward(); optimizer.step() trainer = Trainer(model, args, train_dataset, eval_dataset); trainer.train()— HuggingFace Trainer- Monitoring:
wandb.log({"train_loss": loss.item(), "lr": scheduler.get_last_lr()[0]})— W&B logging - Checkpointing:
torch.save({"epoch": epoch, "model": model.state_dict(), "optim": opt.state_dict()}, "ckpt.pt") - Multi-GPU:
torchrun --nproc_per_node=8 train.py— DDP training on 8 GPUs
Summary
In short: Training is the iterative process of adjusting model weights via gradient descent to minimise loss on training data — the one-time computational investment that encodes all model knowledge, and the most strategically important step in AI development.