← AI Terminology
TensorRT-LLM
TensorRT-LLM is NVIDIA’s high-performance library for optimised LLM inference on NVIDIA GPUs, with kernels for attention, quantisation, and serving features.
It targets maximum throughput/latency on NVIDIA hardware.
It targets maximum throughput/latency on NVIDIA hardware.
Why It Matters in AI
Production GPU serving often needs hand-optimised engines beyond eager PyTorch. TensorRT-LLM compiles models into fast runtimes with in-flight batching and low-precision support — a top choice on NVIDIA stacks.
Key Points
| Aspect | Description |
|---|---|
| Use | Latency-sensitive enterprise inference |
| Vendor | NVIDIA |
| Related | TensorRT, Triton, CUDA |
| Vs vLLM | Often faster on NVIDIA when tuned; less universal/simple |
| Features | FP8/INT4 paths, paged KV, in-flight batching |
| Workflow | Build engine from HF weights → serve |
Simple Analogy
A factory-tuned race ECU for NVIDIA tracks — peak performance when you commit to that hardware platform.
Common Usage Examples
- Build TRT-LLM engines for Llama models
- Serve via Triton backends
- Benchmark vs vLLM on same GPU
- Enable FP8 for throughput
Summary
In short: TensorRT-LLM is NVIDIA’s optimised LLM inference stack — peak GPU serving performance for production NVIDIA deployments.