← AI Terminology
Streaming Response
Streaming response delivers tokens (or events) to the client as they are generated instead of waiting for the full completion.
Implemented via SSE, WebSockets, or chunked HTTP in almost every LLM API.
Implemented via SSE, WebSockets, or chunked HTTP in almost every LLM API.
Why It Matters in AI
Full-wait UX feels broken for long answers. Streaming improves perceived latency, enables cancel mid-flight, and fits token-by-token decode. Clients must handle partial JSON and tool-call edges carefully.
Key Points
| Aspect | Description |
|---|---|
| UX | Lower perceived wait after TTFT |
| APIs | OpenAI stream=True, Anthropic streaming |
| Cancel | Abort generation to save cost |
| Related | TTFT, ITL, autoregressive generation |
| Transport | Server-Sent Events, websockets, gRPC streams |
| Complexity | Partial tool calls; buffered UI rendering |
Simple Analogy
Watching subtitles appear word by word during a talk rather than waiting for the entire transcript file at the end.
Common Usage Examples
for chunk in client.stream(...)- SSE parsers in web apps
- Token buffers for smoother UI
- User hit Stop → cancel server task
Summary
In short: Streaming responses send tokens as they generate — essential UX for interactive LLM applications.