← AI Terminology
GPU - Graphics Processing Unit
A GPU (Graphics Processing Unit) is a massively parallel processor originally designed for rendering graphics, now the dominant hardware for training and running AI models due to its thousands of cores optimised for matrix and tensor operations.
NVIDIA GPUs — particularly the A100 and H100 — are the compute substrate of modern AI.
NVIDIA GPUs — particularly the A100 and H100 — are the compute substrate of modern AI.
Why It Matters in AI
Neural network training is fundamentally matrix multiplication at scale. A CPU has 8–128 high-clock-speed cores; an NVIDIA H100 has 16,896 CUDA cores running in parallel. The H100 delivers 3.958 PFLOP/s of dense FP16 compute — roughly 1000× a high-end CPU for AI workloads. Access to sufficient GPU compute separates who can train frontier AI models from who cannot. GPU availability and pricing directly determines the cost of AI development.
Key Points
| Aspect | Description |
|---|---|
| Memory | HBM (High Bandwidth Memory): H100 SXM: 80 GB HBM3, 3.35 TB/s bandwidth — critical bottleneck |
| Key GPUs | NVIDIA: H100 (AI training), A100 (training/inference), L40S (inference), RTX 4090 (consumer) |
| CUDA cores | Thousands of simpler cores executing the same instruction in parallel — SIMT architecture |
| Investment | NVIDIA's GPU revenue reached $90B/yr in 2024 — the primary beneficiary of AI compute demand |
| Interconnect | NVLink: H100-to-H100 bandwidth 900 GB/s — enables efficient multi-GPU model parallelism |
| AMD alternative | AMD Instinct MI300X: 192 GB HBM3 — competitive for inference; ROCm software stack |
Simple Analogy
A CPU is a team of 8 expert specialists — each can handle complex, varied tasks but can only work on one thing at a time. A GPU is a factory floor with 10,000 workers doing simple, identical operations simultaneously. Matrix multiplication is a factory task — perfect for GPUs; complex branching logic is a specialist task — better for CPUs.
Common Usage Examples
torch.cuda.is_available()— check if GPU is available;tensor.to('cuda')— move data to GPUnvidia-smi— monitor GPU utilisation, memory, temperature, and power- Renting GPUs: AWS p4d.24xlarge (8× A100), Lambda Labs H100 cluster, RunPod, CoreWeave
CUDA_VISIBLE_DEVICES=0,1— restrict training to specific GPU indices- H100 SXM benchmark: trains GPT-3 175B in ~2 weeks with 1024× H100s (vs years on CPUs)
Summary
In short: GPUs are the engine of modern AI — their massively parallel architecture makes matrix operations 1000× faster than CPUs, and access to enough of them determines who can build frontier AI.