← AI Terminology
Next-Token Prediction
Next-token prediction is the self-supervised objective of predicting the probability distribution over the vocabulary for the token immediately following a context.
It is the pretraining task that turned transformers into large language models.
It is the pretraining task that turned transformers into large language models.
Why It Matters in AI
A single simple objective on internet-scale text produces models that translate, code, reason, and follow instructions — no task labels required. Scaling next-token prediction is the central recipe of the GPT lineage.
Key Points
| Aspect | Description |
|---|---|
| Loss | Cross-entropy between predicted distribution and true next token |
| Limits | Does not directly optimise truthfulness — hence SFT/RLHF |
| Variants | Multi-token prediction, fill-in-the-middle, masked LM |
| Emergence | Broad abilities arise as scale increases |
| Eval proxy | Perplexity measures next-token surprise on held-out text |
| Self-supervised | Labels are free — text itself is the supervision |
Simple Analogy
Reading a book with the next word covered and constantly guessing it — after millions of pages, you become eerily good at language.
Common Usage Examples
- Pretrain:
loss = CrossEntropy(logits[:, :-1], tokens[:, 1:]) - GPT/LLaMA pretraining objective
- Perplexity = exp(average NLL)
- FIM: predict middle span given prefix+suffix
Summary
In short: Next-token prediction is the deceptively simple objective — guess the next piece of text — that scales into general-purpose large language models.