← AI Terminology
Cross-Entropy
Cross-entropy is a loss function that measures how different a predicted probability distribution is from the true distribution — returning higher values when confident predictions are wrong and lower values when they are correct.
It is the standard training objective for classification models and language models.
It is the standard training objective for classification models and language models.
Why It Matters in AI
Cross-entropy is the most common loss in deep learning because it derives from maximum likelihood estimation: minimising cross-entropy is equivalent to maximising the probability the model assigns to the correct answer. For language models, minimising cross-entropy over next-token prediction is the entire pre-training objective — every GPT, Llama, and Claude was trained by minimising it.
Key Points
| Aspect | Description |
|---|---|
| Binary CE | H = −[y log(p) + (1−y) log(1−p)] — for binary classification |
| Perplexity | exp(cross-entropy) — interpretable form: how many equally likely options the model "sees" |
| Relationship | Cross-entropy = KL divergence from true to predicted distribution + entropy of true distribution |
| Categorical CE | H = −Σ y_i log(p_i) — for multiclass; most deep learning classifiers use this |
| Language models | H = −(1/T) Σ log P(token_t |
| Numerical stab. | Applied to logits before softmax: log_softmax + NLLLoss = CrossEntropyLoss in PyTorch |
Simple Analogy
Cross-entropy is the cost of encoding a message: if you predict "definitely A" but the answer is B, you pay a massive cost (−log(0.01) ≈ 4.6). If you predict "probably A (60%)" and it is A, you pay less (−log(0.6) ≈ 0.5). Training minimises the total encoding cost — forcing the model to put high probability on correct answers.
Common Usage Examples
nn.CrossEntropyLoss()in PyTorch — expects raw logits, applies softmax internally- LLM training loss:
loss = F.cross_entropy(logits.view(-1, vocab_size), labels.view(-1)) - Temperature scaling:
logits / Tbefore softmax — lowers/raises effective cross-entropy - Label smoothing:
CrossEntropyLoss(label_smoothing=0.1)— prevents overconfident predictions - GPT-3 training perplexity ~20 on validation set → exp(20/T) depends on token count
Summary
In short: Cross-entropy is the loss function that teaches classifiers and language models to assign high probability to the right answer — the universal training objective in deep learning.