← AI Terminology
Autoregressive Generation
Autoregressive generation produces a sequence step by step, where each new token is sampled conditioned on all previously generated tokens.
It is the dominant decoding paradigm for LLMs, many speech codecs, and sequential multimodal generators.
It is the dominant decoding paradigm for LLMs, many speech codecs, and sequential multimodal generators.
Why It Matters in AI
Language is sequential; autoregression turns next-token prediction into open-ended generation. Quality, latency, and serving tricks (KV cache, speculative decoding) all orbit this left-to-right loop.
Key Points
| Aspect | Description |
|---|---|
| Cost | Serial decode steps; prefill is parallel, decode is sequential |
| Loop | Context → sample token → append → repeat until EOS/limit |
| KV cache | Reuses past K/V so each step avoids full recompute |
| Sampling | Greedy, temperature, top-k, top-p, beam search |
| Alternatives | Non-autoregressive models, diffusion LMs, masked parallel decoding |
| Teacher forcing | Training feeds gold previous tokens; inference feeds its own |
Simple Analogy
Finishing a sentence one word at a time, always reading everything written so far before choosing the next word.
Common Usage Examples
model.generate(**inputs, max_new_tokens=256)— Hugging Face- OpenAI/Anthropic APIs stream tokens autoregressively
- vLLM continuous batching of many AR sequences
- Stop on EOS or custom stop strings
Summary
In short: Autoregressive generation builds output token-by-token, each conditioned on the past — the core loop behind virtually all modern LLM text generation.