← AI Terminology
Constrained Decoding
Constrained decoding restricts the tokens a model may emit at each step so outputs obey a grammar, regex, or finite automaton (e.g. always-valid JSON or a domain-specific language).
It is the inference-time engine behind reliable structured generation.
It is the inference-time engine behind reliable structured generation.
Why It Matters in AI
Sampling freely often breaks formats. Masking illegal tokens using a grammar gives hard guarantees without fragile re-prompting. Open-source stacks (Outlines, Guidance, llama.cpp grammars) and cloud 'strict schema' modes rely on it.
Key Points
| Aspect | Description |
|---|---|
| Con | Implementation complexity; possible quality tradeoffs |
| Pro | Guaranteed syntactic validity |
| Specs | JSON Schema, regex, context-free grammars, type systems |
| Tools | Outlines, Guidance, LMQL, llama.cpp GBNF |
| Related | Structured outputs, function argument filling |
| Mechanism | At each step, mask logits outside the allowed next-token set |
Simple Analogy
An autocomplete that only offers words that keep your sentence grammatically legal in a formal language — illegal next words are simply not choosable.
Common Usage Examples
- GBNF grammars in llama.cpp
outlines.generate.json(model, schema)- Strict tool-argument decoding
- SQL/DSL generation under grammar
Summary
In short: Constrained decoding masks illegal tokens so generations are guaranteed to follow a grammar or schema.