← AI Terminology
Continuous Batching
Continuous batching (also called in-flight batching) is an LLM serving technique where new requests are dynamically added to a batch mid-generation, rather than waiting for all current requests to finish before starting the next batch.
It dramatically improves GPU utilisation and throughput in production LLM deployments.
It dramatically improves GPU utilisation and throughput in production LLM deployments.
Why It Matters in AI
Traditional batching waits for the slowest request in a batch to finish before starting new ones — a short query is blocked by a long one running alongside it. Continuous batching slots completed requests out and new ones in at every token generation step, keeping the GPU fully loaded. vLLM, TGI, and SGLang all implement it, and it is responsible for the order-of-magnitude throughput improvements in modern LLM serving stacks.
Key Points
| Aspect | Description |
|---|---|
| Mechanism | At each decode step, finished sequences are replaced by new requests — batch composition changes continuously |
| Key systems | vLLM, HuggingFace TGI, NVIDIA Triton + TensorRT-LLM, SGLang |
| Problem solved | Static batching: short requests idle waiting for long requests in same batch to finish |
| Throughput gain | Typically 10–23× throughput improvement over naïve static batching (Orca paper, 2022) |
| Memory management | Paired with PagedAttention (vLLM) to efficiently manage KV cache for dynamically-sized sequences |
| Prefill vs decode | Prefill (processing input) is compute-bound; decode (generating tokens) is memory-bandwidth-bound |
Simple Analogy
An old restaurant where tables must all finish before new diners are seated (static batching) vs. a modern restaurant where tables turn over individually — as soon as a party leaves, the table is reset and new guests are seated. Continuous batching runs the kitchen at capacity rather than waiting for the slowest table.
Common Usage Examples
- vLLM:
vllm serve meta-llama/Meta-Llama-3-8B-Instruct— continuous batching enabled by default - HuggingFace TGI:
--max-batch-prefill-tokens 4096— controls continuous batching parameters - Measured benchmark: Llama-2 70B — continuous batching achieves ~23× throughput vs static
- Production API: OpenAI, Anthropic, and Google all use continuous batching on their inference clusters
- SGLang: continuous batching + RadixAttention for further KV cache reuse across requests
Summary
In short: Continuous batching keeps GPUs fully loaded by dynamically swapping finished requests out and new ones in mid-generation — the technique that made scalable LLM APIs economically viable.