← AI Terminology
Speculative Decoding
Speculative decoding is an LLM inference optimisation that uses a small, fast "draft" model to speculatively generate multiple tokens ahead, then verifies all of them in a single forward pass of the large target model — accepting correct tokens and regenerating from the first mismatch, achieving 2–4× speedup with identical output distribution.
It decouples token generation speed from model size.
It decouples token generation speed from model size.
Why It Matters in AI
LLM inference is memory-bandwidth-bound: each forward pass reads the full model weights to produce one token — 7B model ≈ 14GB read to produce 10 bytes. GPU memory bandwidth is fixed; utilisation per token is ~1%. Speculative decoding batches multiple token verifications into one pass: the draft model produces 4–8 speculative tokens cheaply; one target model forward pass verifies all of them; accepted tokens appear "for free" relative to sequential generation. This is one of the largest practical inference speedups available.
Key Points
| Aspect | Description |
|---|---|
| EAGLE | Improved self-speculation: 3× speedup using feature-level draft prediction |
| Speedup | 2–4× on long-form generation — speedup proportional to acceptance rate × draft tokens per round |
| Draft model | Small model (e.g. 7B target + 70M draft) or self-speculation (same model, earlier layers) |
| Acceptance rate | Fraction of draft tokens accepted — depends on draft/target alignment; typically 70–85% |
| Output identical | Mathematically equivalent to standard sampling — no quality tradeoff |
| Self-speculation | Medusa: single model with multiple prediction heads — no external draft model needed |
Simple Analogy
Drafting a document in rough shorthand (draft model) then having an expert editor verify it in one pass (target model forward pass): the editor accepts most sentences as-is and only corrects clear mistakes. Instead of the editor writing every word from scratch, they verify and lightly edit — far faster than drafting from scratch.
Common Usage Examples
- vLLM:
--speculative-model="facebook/opt-125m"— draft model for speculative decoding - HuggingFace:
model.generate(input_ids, assistant_model=draft_model)— built-in speculative decoding - Medusa heads:
from medusa.model.medusa_model import MedusaModel— self-speculation without draft model - EAGLE:
from eagle.model.ea_model import EaModel— 3× speedup via feature-level speculation - Google JAX:
speculative_decode(draft_model, target_model, input_ids, num_speculative=5)— cloud deployment
Summary
In short: Speculative decoding uses a small draft model to speculatively generate multiple tokens that a large target model verifies in one pass — achieving 2–4× inference speedup with mathematically identical output, one of the most practical LLM serving optimisations.