← AI Terminology
Inference
Inference is the process of running a trained AI model on new input data to produce predictions, classifications, or generated content — distinct from training, which is the process of learning the model's weights.
Inference is what end users experience: every ChatGPT response, image generation, and recommendation is an inference call.
Inference is what end users experience: every ChatGPT response, image generation, and recommendation is an inference call.
Why It Matters in AI
Training cost is a one-time expense; inference cost is recurring and scales with usage. A model serving millions of requests per day accumulates far more total compute than its training run. Inference optimisation — quantisation, speculative decoding, continuous batching, distillation — directly determines the economics and latency of AI products. Most AI companies now spend more on inference than training infrastructure.
Key Points
| Aspect | Description |
|---|---|
| Latency | Time to first token (TTFT) + time per output token (TPOT) — key UX metrics for LLMs |
| Batch size | Larger batches → higher throughput; smaller batches → lower latency — tradeoff |
| Throughput | Tokens per second across all concurrent users — determines cost per million tokens |
| Cloud vs edge | Cloud inference: scalable, expensive; edge inference (phone, laptop): private, latency-free |
| Optimisations | Quantisation (INT8/FP8), KV cache, Flash Attention, continuous batching, speculative decoding |
| Serving infra | vLLM, TGI, TensorRT-LLM, Triton Inference Server — inference-specialised serving frameworks |
Simple Analogy
A trained chef (model) applying their skills to cook meals for customers (inference) — training was culinary school (learning). The chef doesn't re-learn to cook each time; they apply existing skills instantly. The restaurant's throughput (meals/hour) and speed (time to serve one meal) determine the business economics — that's the inference optimisation problem.
Common Usage Examples
model.eval(); with torch.no_grad(): outputs = model(input_tensor)— PyTorch inference modemodel.generate(input_ids, max_new_tokens=200, do_sample=True)— LLM text generation- ONNX export:
torch.onnx.export(model, sample_input, "model.onnx")— optimised inference format - TensorRT:
trt.Runtimewith FP16 precision — 2–4× faster inference on NVIDIA GPUs - vLLM:
python -m vllm.entrypoints.openai.api_server --model meta-llama/Meta-Llama-3-8B
Summary
In short: Inference is the moment the model actually works — taking input and producing output — and optimising it is the central engineering challenge for making AI products fast and economical at scale.