← AI Terminology
TensorRT
TensorRT is NVIDIA's SDK for high-performance deep learning inference — it compiles trained models into optimised CUDA engines through layer fusion, precision calibration (FP16/INT8), and kernel auto-selection, typically achieving 2–6× speedup over framework-native inference on NVIDIA GPUs.
It is the production inference optimiser for NVIDIA hardware.
It is the production inference optimiser for NVIDIA hardware.
Why It Matters in AI
A PyTorch model exported to production uses framework-level inference: each operation is a separate kernel call, precision is FP32, and there is no cross-layer optimisation. TensorRT analyses the full computation graph, fuses operations into single kernels (Conv + BN + ReLU → one kernel), selects optimal CUDA implementations for each shape, and calibrates INT8 quantisation to preserve accuracy. The result is typically 2–6× higher throughput with lower latency — directly reducing GPU cost for production inference.
Key Points
| Aspect | Description |
|---|---|
| Engine | Compiled .trt or .engine file — hardware-specific, must recompile for different GPU types |
| Precision | FP32 → FP16 → INT8 compilation — INT8 calibration requires representative dataset |
| ONNX input | trtexec --onnx=model.onnx — standard conversion path: framework → ONNX → TensorRT |
| Integration | TensorRT-LLM: NVIDIA's LLM-optimised TRT library with in-flight batching and PagedAttention |
| Layer fusion | Conv + BatchNorm + Activation compiled into one optimised kernel — eliminates memory round-trips |
| Dynamic shapes | --minShapes, --optShapes, --maxShapes — handle variable batch sizes at inference time |
Simple Analogy
A racing car tuner: the stock car (PyTorch model) runs fine but is not optimised for the track (NVIDIA GPU). TensorRT is the tuner who strips unnecessary components, fits the engine to the track conditions, and adjusts every parameter for peak performance — producing a race-ready version that goes 4× faster on the same engine.
Common Usage Examples
trtexec --onnx=model.onnx --saveEngine=model.trt --fp16— compile ONNX model to TRT FP16 enginetrtexec --onnx=model.onnx --int8 --calib=calib_data.npy— INT8 calibration- Python:
import tensorrt as trt; engine = runtime.deserialize_cuda_engine(f.read()) context = engine.create_execution_context(); context.execute_v2(bindings=[d_input, d_output])- TensorRT-LLM:
trtllm-build --checkpoint-dir ./llama3-fp8 --output-dir ./engine --dtype fp8
Summary
In short: TensorRT compiles trained models into optimised CUDA engines through layer fusion and precision calibration — delivering 2–6× inference speedup on NVIDIA GPUs, the standard production optimisation step for NVIDIA-based AI deployment.