← AI Terminology
Beam Search
Beam search is a heuristic search algorithm that generates sequences (text, translations, speech) by maintaining the top-k most probable partial sequences ("beams") at each step rather than committing to a single best token.
It is the standard decoding algorithm for sequence-to-sequence models when quality matters more than speed.
It is the standard decoding algorithm for sequence-to-sequence models when quality matters more than speed.
Why It Matters in AI
Greedy decoding picks the single highest-probability token at each step — fast but often produces suboptimal sequences because a high-probability first token can lead to a poor overall sequence. Beam search explores multiple paths simultaneously, producing noticeably better translations, summaries, and captions. Most production NLP systems (Google Translate, ASR systems) use beam search at inference time.
Key Points
| Aspect | Description |
|---|---|
| Diversity | Diverse beam search forces beams to differ — avoids near-identical output candidates |
| LLM usage | Most frontier LLMs use sampling (temperature/top-p) for generation, not beam search |
| Exploration | Keeps k candidate sequences; at each step expands all k × vocab candidates and prunes to best k |
| vs sampling | Beam search maximises probability; sampling (top-p) introduces randomness — preferred for LLMs |
| Beam width k | k=1 is greedy decoding; k=4–6 is typical for NLP; larger k = better but slower |
| Length penalty | Without it, beam search favours shorter sequences (higher probability product) — add α penalty |
Simple Analogy
Navigating a maze: greedy navigation always takes the widest corridor. Beam search simultaneously explores the 4 widest corridors, discarding dead ends and keeping the most promising paths — arriving at a better exit route than any single greedy choice would reach.
Common Usage Examples
model.generate(input_ids, num_beams=4, length_penalty=1.0)in HuggingFace Transformers- Google Translate uses beam search with beam width 4–8 for translation
- Whisper ASR:
model.transcribe(audio, beam_size=5)for higher-quality transcription - Diverse beam search:
num_beam_groups=4, diversity_penalty=0.5in HuggingFace - Code generation: beam search used to return top-k distinct code completions (GitHub Copilot)
Summary
In short: Beam search generates better sequences than greedy decoding by tracking multiple candidate paths simultaneously — the quality/speed tradeoff is controlled by the beam width.