← AI Terminology
Top-p / Top-k Sampling
Top-k sampling limits token selection to the k most probable candidates at each step; top-p (nucleus) sampling selects from the smallest set of tokens whose cumulative probability exceeds p — both methods truncate the probability distribution to avoid very low-probability (incoherent) token selections while maintaining diversity.
They are the two primary methods for controlling generation diversity in LLMs.
They are the two primary methods for controlling generation diversity in LLMs.
Why It Matters in AI
Greedy decoding (always pick the most probable token) produces repetitive, boring text. Sampling from the full vocabulary risks generating random, incoherent tokens with tiny probabilities. Top-k and top-p find the middle ground: enough candidates for diversity, few enough to exclude clearly wrong tokens. Top-p is generally preferred over top-k because it adapts to the distribution shape — when the model is confident, only 1–2 tokens qualify; when uncertain, many more are included.
Key Points
| Aspect | Description |
|---|---|
| Top-k | Keep only the k most probable tokens; sample from those — k=1 = greedy; k=50 = common default |
| Top-p | Keep smallest set summing to probability p; sample from those — p=0.9 or 0.95 most common |
| min-p | Newer alternative: min_p=0.05 filters tokens below 5% of max probability — simpler, effective |
| Combined use | Many systems apply both: top-p=0.9 AND top-k=50 — apply whichever is more restrictive first |
| Adaptive nature | Top-k is fixed; top-p adapts to distribution: confident model → small nucleus; uncertain → large |
| Temperature first | Temperature is applied to logits before top-p/top-k filtering — order matters |
Simple Analogy
A restaurant menu with hundreds of options: top-k is "only consider the 50 most popular dishes"; top-p is "only consider dishes until you've accounted for 90% of what customers normally order." Both prevent you from randomly ordering the one dish nobody ever chooses — but top-p adapts when the restaurant has a few very popular dishes vs. many equally popular ones.
Common Usage Examples
model.generate(input_ids, do_sample=True, top_k=50, top_p=0.95, temperature=0.8)— HuggingFace- OpenAI:
client.chat.completions.create(model="gpt-4o", top_p=0.9)— top-p only (OpenAI recommends not mixing) - Anthropic:
client.messages.create(model="claude-opus-4-7", top_p=0.95, top_k=50)— both supported - min-p:
model.generate(input_ids, min_p=0.05)— filter tokens below 5% of max token probability - Deterministic:
temperature=0.0, do_sample=False— disable both, use greedy decoding
Summary
In short: Top-k limits candidates to the k most probable tokens; top-p selects the smallest set summing to probability p — both prevent incoherent low-probability token selection while maintaining diversity, with top-p preferred for its adaptive behavior across different distribution shapes.