← AI Terminology
Perplexity
Perplexity is an intrinsic language model evaluation metric that measures how well a model predicts a held-out text corpus — defined as the exponent of the average cross-entropy loss per token: lower perplexity means the model assigns higher probability to the actual next tokens.
It is the standard unsupervised evaluation metric for language models.
It is the standard unsupervised evaluation metric for language models.
Why It Matters in AI
Perplexity provides a fast, automatic measure of language model quality without human annotation: lower perplexity means the model is better at predicting real text. It is used to compare pre-training runs, evaluate quantisation degradation, and track training progress. However, it only measures in-distribution text prediction — a model with low perplexity on Wikipedia may still hallucinate, fail at reasoning, or give harmful outputs. MMLU and human preference evaluations complement perplexity.
Key Points
| Aspect | Description |
|---|---|
| Range | [1, ∞) — 1 = perfect prediction; random token predictor on 50K vocab ≈ 50,000 |
| Formula | PPL = exp(-(1/n) × Σ log P(wᵢ |
| Quantisation | Perplexity is the standard metric for measuring quantisation degradation (Q4 vs Q8 vs FP16) |
| Interpretation | PPL of 20 ≈ the model is as uncertain as choosing uniformly among 20 options at each step |
| Sliding window | For sequences longer than context: stride through with overlapping windows, aggregate PPL |
| Comparison caveat | Only comparable across models with the same tokenizer/vocabulary — different vocab = different PPL |
Simple Analogy
A reading comprehension score where the student predicts the next word: a student who rarely guesses correctly (high perplexity) understands the text less well; one who consistently predicts what comes next (low perplexity) has deeply internalised the language patterns.
Common Usage Examples
lm_eval --tasks wikitext --model hf --model_args pretrained=llama3— standard perplexity evaluationloss = model(input_ids, labels=input_ids).loss; ppl = torch.exp(loss)— PyTorch perplexity- GGUF quantisation comparison: Q4_K_M perplexity vs Q8_0 — measure quality cost of quantisation
- Sliding window PPL:
from transformers import evaluate; perplexity = evaluate.load("perplexity") - LLM pre-training tracking: perplexity on held-out validation set logged every 1,000 steps
Summary
In short: Perplexity measures how well a language model predicts held-out text — the standard intrinsic quality metric for pre-training and quantisation evaluation, where lower is better, though it must be complemented by task-based benchmarks for real-world quality assessment.