← AI Terminology
Nucleus Sampling
Nucleus sampling (top-p sampling) is a text generation strategy that at each step samples the next token from only the smallest subset of the vocabulary whose cumulative probability exceeds a threshold p — dynamically adapting the candidate set size to the model's confidence, unlike top-k which uses a fixed number of candidates.
It was proposed by Holtzman et al. (2020) and is now the standard LLM sampling method.
It was proposed by Holtzman et al. (2020) and is now the standard LLM sampling method.
Why It Matters in AI
Top-k sampling selects from a fixed number of tokens regardless of their probability distribution — sometimes including many unlikely candidates (when the distribution is flat) or excluding decent options (when it's peaked). Nucleus sampling adapts: with p=0.9, if the model is confident (one token has 95% probability), the nucleus has one token; if uncertain, it includes enough tokens to reach 90% cumulative probability. This produces more coherent text than top-k while remaining more creative than greedy decoding.
Key Points
| Aspect | Description |
|---|---|
| vs top-k | Top-k: always k candidates; nucleus: variable candidates depending on distribution shape |
| Algorithm | Sort tokens by descending probability → accumulate until sum ≥ p → sample from this nucleus |
| Combination | Most APIs use temperature + nucleus sampling together — temperature shapes; nucleus limits floor |
| Temperature | Applied before nucleus selection: divides logits, sharpening (T<1) or flattening (T>1) probabilities |
| p parameter | Typical range 0.8–0.95 — lower = less diversity; higher = more diversity; 1.0 = full vocabulary |
| Greedy decoding | p=0 effectively → always pick argmax — deterministic but often repetitive |
Simple Analogy
A DJ who doesn't have a fixed playlist length, but plays songs until they've covered 90% of what the crowd wants to hear (based on requests). If the crowd is unanimous about one song (confident model), that's all that plays. If tastes are split, many songs get added to reach 90% coverage. Nucleus sampling adapts to the crowd's certainty.
Common Usage Examples
- OpenAI API:
client.chat.completions.create(model="gpt-4o", top_p=0.95, temperature=0.7) - HuggingFace:
model.generate(input_ids, do_sample=True, top_p=0.9, temperature=0.8) top_p=0.9, temperature=1.0— common creative writing defaultstop_p=0.1, temperature=0.2— factual QA defaults (near-deterministic)- Anthropic API:
client.messages.create(model="claude-opus-4-7", top_p=0.95, temperature=1.0)
Summary
In short: Nucleus sampling selects the next token from the smallest set of candidates whose probabilities sum to p — dynamically adapting the candidate pool to the model's confidence, producing more coherent and creative text than fixed top-k sampling.