← AI Terminology
Tensor Core
Tensor Cores are specialised processing units in NVIDIA GPUs (Volta and later) designed exclusively for matrix multiplication operations — performing mixed-precision matrix multiply-accumulate (MMA) operations, delivering 4–16× higher throughput than standard CUDA cores for deep learning workloads.
They are the hardware reason modern AI training is economically feasible.
They are the hardware reason modern AI training is economically feasible.
Why It Matters in AI
Training a neural network is dominated by matrix multiplications (attention, linear layers, convolutions). Tensor Cores execute these in hardware with dedicated silicon — an A100 has 432 Tensor Cores delivering 312 TFLOPS FP16 vs 19.5 TFLOPS FP32 on CUDA cores (16× speedup). This performance gap is why FP16/BF16 training is universal: the speedup is so large (from training weeks to days) that the precision trade-off is always worth it. Every modern AI accelerator (Google TPU, AWS Trainium, AMD Tensor Engine) uses an equivalent specialised MMA unit.
Key Points
| Aspect | Description |
|---|---|
| Operation | D = A×B + C — 4×4 matrix multiply-accumulate in a single instruction cycle |
| Precision | FP16/BF16 inputs, FP32 accumulation — mixed precision for speed + stability |
| Activation | torch.set_float32_matmul_precision("high") — enable TF32 Tensor Core use for FP32 matmuls |
| Throughput | H100 SXM: 989 TFLOPS FP16 (Tensor Core) vs 66.9 TFLOPS FP32 (CUDA Core) — 14.8× difference |
| FP8 (Hopper) | H100 adds FP8 Tensor Cores — 1,979 TFLOPS — further 2× for inference |
| cuBLAS / cuDNN | All linear layers and convolutions automatically route to Tensor Cores via these libraries |
Simple Analogy
A specialised assembly line vs. a general factory floor: a general factory (CUDA cores) can make anything, slowly; a matrix-multiplication assembly line (Tensor Cores) does one thing at 16× the speed. Since AI training is 90%+ matrix multiplications, having that dedicated line changes everything.
Common Usage Examples
torch.backends.cuda.matmul.allow_tf32 = True— enables TF32 Tensor Core for FP32 matmulsmodel.half()ortorch.autocast("cuda")— moves ops to FP16 Tensor Core path automaticallyTORCH_CUDNN_V8_API_ENABLED=1— enable cuDNN v8 attention using Tensor Core Flash Attention- A100 vs V100: A100 has 3rd-gen Tensor Cores with BF16 support — factor in LLM training choice
nvidia-smi dmon+ Nsight: monitor Tensor Core utilisation — high SM active = Tensor Core saturated
Summary
In short: Tensor Cores are NVIDIA's dedicated matrix-multiply silicon — delivering 4–16× the FP16 throughput of general CUDA cores, making them the hardware enabler of economically feasible large-scale AI training and the reason mixed-precision training is universally adopted.