← AI Terminology
Loss Function
A loss function (also called cost function or objective function) is a mathematical function that quantifies how wrong a model's predictions are — producing a scalar value that gradient descent minimises to train the model.
The choice of loss function defines what the model optimises for.
The choice of loss function defines what the model optimises for.
Why It Matters in AI
The loss function is the compass of training: it tells the optimiser which direction to move weights. Choosing the wrong loss function (e.g. MSE for classification) trains a model that technically minimises the function but doesn't solve the intended problem. Loss functions must be differentiable (so gradients can be computed) and must align with the actual goal — misalignment between loss and goal is a root cause of AI systems that behave unexpectedly in production.
Key Points
| Aspect | Description |
|---|---|
| Regression | Mean Squared Error (MSE) or Mean Absolute Error (MAE) — for predicting continuous values |
| Classification | Cross-entropy loss (log-loss) — standard for classification tasks |
| Ranking / RLHF | Contrastive losses, pairwise ranking loss — for preference learning and reward modelling |
| Language models | Cross-entropy on next-token prediction — averaged over all output tokens |
| Object detection | Combination: classification loss + bounding box regression loss (e.g. GIoU loss) |
| Differentiability | Loss must be differentiable (or subgradient-differentiable) for gradient-based optimisation |
Simple Analogy
A golf scorecard: the score (loss) summarises how far from the hole each shot landed. The golfer (optimiser) adjusts their swing (weights) to lower the score. A scorecard that only tracked whether the ball went right or left (wrong loss) would train a golfer who is perfectly centred but shoots miles past the hole.
Common Usage Examples
loss = nn.CrossEntropyLoss()(logits, targets)— multiclass classification in PyTorchloss = nn.MSELoss()(predictions, targets)— regression lossloss = nn.BCEWithLogitsLoss()(logits, targets)— binary classification with numerical stability- LLM training:
loss = cross_entropy(model_logits, token_ids, ignore_index=pad_token_id) - RLHF reward model:
loss = -log(sigmoid(reward_chosen - reward_rejected))— Bradley-Terry pair ranking
Summary
In short: The loss function is the objective that training minimises — it defines what "better" means and must be carefully chosen to align with the actual problem, or the model will optimise for the wrong thing.