← AI Terminology
Search at Inference
Search at inference uses deliberate exploration algorithms (beam search, MCTS, tree-of-thoughts expansion, tool-using plan search) at decode time to find better outputs than single-pass generation.
It combines generators with scoring and branching.
It combines generators with scoring and branching.
Why It Matters in AI
Greedy or single-sample decode underuses the model. Explicit search expands candidates and backtracks using rewards or heuristics — key to reasoning systems, code synthesis, and game-playing AI.
Key Points
| Aspect | Description |
|---|---|
| Cost | Multiplies inference FLOPs; needs pruning |
| Classic | AlphaGo-style MCTS; NLP beam search |
| LLM era | Tree-of-thoughts, graph-of-thoughts, rStar |
| Guidance | Logprobs, PRMs, unit tests, heuristics |
| Tradeoff | Latency vs quality; parallel branches help |
| Algorithms | Beam search, MCTS, BFS/DFS over thoughts, A* variants |
Simple Analogy
Not walking a single path through a maze, but exploring several corridors, marking dead ends, and committing when a route looks best.
Common Usage Examples
- Beam search in MT and coding models
- MCTS over reasoning steps with a PRM
- Tree-of-thoughts prompting with branch evaluation
- Program synthesis with test-guided search
Summary
In short: Search at inference explores multiple candidate paths with scoring and backtracking — trading compute for better answers than a single greedy decode.