← AI Terminology
Dynamic Batching
Dynamic batching groups inference requests that arrive close in time into a batch to raise GPU utilisation, with a latency cap so early requests do not wait forever.
Classic model-server technique; LLMs extend it with continuous batching.
Classic model-server technique; LLMs extend it with continuous batching.
Why It Matters in AI
GPUs love batches; users hate waits. Dynamic batching balances throughput and latency for embeddings, classifiers, and older seq2seq servers. Understanding it leads into continuous batching for LLMs.
Key Points
| Aspect | Description |
|---|---|
| Cost | Added queuing latency |
| Knob | Max batch size + max wait window |
| Benefit | Higher throughput and lower $/query |
| Classic | Triton dynamic batcher |
| Related | Model serving, continuous batching |
| LLM evolution | Continuous/in-flight batching (vLLM) |
Simple Analogy
An elevator waiting a short moment to fill with passengers before moving — better capacity, slight wait for the first person in.
Common Usage Examples
- Triton
dynamic_batchingconfig - Embedding service micro-batches
- Tune max_queue_delay_microseconds
- Compare QPS vs p95 latency curves
Summary
In short: Dynamic batching groups nearby requests to fill the GPU — trading a little wait for much higher throughput.