← AI Terminology
Tokens Per Second
Tokens per second (TPS / tok/s) measures generation throughput — how many tokens the system emits per unit time for a request or across a server.
It is the headline decode-speed metric for inference.
It is the headline decode-speed metric for inference.
Why It Matters in AI
After the first token, UX depends on stream rate. Hardware, batching, quantisation, and speculative decoding all target TPS. Compare single-stream TPS vs aggregate server TPS carefully.
Key Points
| Aspect | Description |
|---|---|
| Caveat | Longer contexts slow decode (KV growth) |
| Related | ITL, continuous batching, vLLM |
| Boosters | Quantisation, GQA, speculative decoding, better kernels |
| Bound by | Memory bandwidth for decode; compute for prefill |
| Aggregate | Total tokens/s across concurrent requests |
| Per-request | Interactive stream speed for one user |
Simple Analogy
How quickly a live captioner types words once they have started — the ongoing pace of the reply.
Common Usage Examples
- Benchmark
tok/son fixed GPUs - Compare FP16 vs INT4 TPS
- Server dashboard aggregate tokens/s
- Speculative decoding speedups
Summary
In short: Tokens per second measures generation speed — the core throughput number for LLM inference engineering.