← AI Terminology

Inference Engine

An inference engine is a software system optimised for running trained AI models at production scale — handling batching, hardware acceleration, memory management, and serving infrastructure to maximise throughput and minimise latency.

Examples: TensorRT (NVIDIA), vLLM, HuggingFace TGI, ONNX Runtime, llama.cpp.
Why It Matters in AI
A model exported from PyTorch training is not production-ready: it lacks batching, has no KV cache management, and doesn't exploit low-precision kernels. Inference engines are the performance and reliability layer between a trained model and real users. The difference between a naïve PyTorch server and vLLM on the same hardware can be 20–50× throughput improvement — directly determining the cost of serving one million API calls.
Key Points
Aspect Description
TensorRT NVIDIA-specific graph compilation → fused CUDA kernels — best GPU inference throughput
Precision INT8, FP8, FP16 quantised kernels — 2–4× speedup and memory reduction vs FP32
Edge engines llama.cpp (CPU+GPU), ExLlamaV2, MLC LLM — run quantised LLMs on consumer hardware
LLM-specific vLLM (PagedAttention + continuous batching), TGI (Token streaming), SGLang (RadixAttention)
ONNX Runtime Cross-platform inference for vision/NLP models — optimised backends for CPU, GPU, NPU
Key optimisations Operator fusion, kernel auto-tuning, memory planning, continuous batching, KV cache management
Simple Analogy
A race car engine vs a street car engine: both can drive, but the race car engine is tuned for maximum performance — every component optimised, no comfort features, precision calibration. Inference engines do the same to trained models: strip out everything not needed for prediction and optimise what remains for speed and efficiency.
Common Usage Examples
  • vLLM: vllm serve mistralai/Mistral-7B-Instruct-v0.3 --port 8000 — production LLM API server
  • TensorRT: trtexec --onnx=model.onnx --saveEngine=model.trt --fp16 — compile for NVIDIA GPU
  • ONNX Runtime: ort.InferenceSession("model.onnx") — cross-platform CPU/GPU inference
  • llama.cpp: ./llama-cli -m llama-3-8b.Q4_K_M.gguf -p "Hello" — CPU inference on any machine
  • TGI: docker run ghcr.io/huggingface/text-generation-inference --model-id meta-llama/Meta-Llama-3-8B
Summary
In short: An inference engine transforms a trained model into a fast, scalable production system — the layer that determines actual serving cost and latency, and often provides 10–50× performance improvement over raw framework inference.