← AI Terminology
Model Serving
Model serving is the production practice of hosting models behind APIs with batching, scaling, routing, observability, and SLOs — not just running
It is the MLOps/LLMOps discipline of reliable inference.
generate() in a notebook.It is the MLOps/LLMOps discipline of reliable inference.
Why It Matters in AI
Research demos die without serving: autoscaling, multi-model routing, canaries, auth, and cost control. LLM serving adds KV cache, continuous batching, and streaming semantics on top of classic model servers.
Key Points
| Aspect | Description |
|---|---|
| Obs | TTFT, TPS, queue depth, GPU util, errors |
| Stacks | vLLM, TGI, TensorRT-LLM, Triton, SageMaker, KServe |
| Related | LLMOps, continuous batching |
| Concerns | Latency, throughput, cost, reliability, security |
| Patterns | Blue/green, canary, A/B, shadow traffic |
| LLM specifics | Streaming, prefix cache, LoRA multiplex |
Simple Analogy
A restaurant kitchen during dinner rush — not one home meal, but stations, tickets, timing, and quality under load.
Common Usage Examples
- Deploy vLLM OpenAI-compatible server
- Kubernetes GPU deployments
- Canary 5% traffic to new weights
- Dashboards for GPU cache and latency
Summary
In short: Model serving is production hosting of models with scale, SLOs, and ops — the difference between a demo and a product.