← AI Terminology
Inter-Token Latency
Inter-token latency (ITL) is the time between consecutive output tokens during decode (sometimes reported as average or p95 gaps).
It determines how smooth streaming text feels.
It determines how smooth streaming text feels.
Why It Matters in AI
Even with good TTFT, choppy streams feel broken. ITL captures decode regularity under load and helps tune continuous batching and scheduling. Product teams watch ITL alongside TPS.
Key Points
| Aspect | Description |
|---|---|
| UX | Smooth typing effect vs stuttering |
| Report | Mean/p95 ITL over a response |
| Tuning | Stable continuous batching policies |
| Drivers | Batch size changes, preemption, KV bandwidth |
| Related | TPS ≈ 1/mean_ITL for single stream |
| Definition | Δt between token i and i+1 |
Simple Analogy
The gap between keystrokes on a typewriter stream — even spacing feels natural; random pauses feel laggy.
Common Usage Examples
- Trace token timestamps in streaming clients
- Load tests recording ITL histograms
- Tune max batch to protect ITL SLOs
- Compare engines under identical QPS
Summary
In short: Inter-token latency is the pause between streamed tokens — the smoothness metric for live generation.