← AI Terminology
HBM - High Bandwidth Memory
HBM (High Bandwidth Memory) is a 3D-stacked DRAM standard that bonds multiple DRAM dies vertically with through-silicon vias (TSVs) and connects them to the processor via an ultra-wide bus (1,024–2,048 bits) — delivering memory bandwidth an order of magnitude higher than conventional GDDR.
It is the memory technology inside every major AI accelerator: NVIDIA H/B-series, AMD MI-series, Google TPU, Intel Gaudi.
It is the memory technology inside every major AI accelerator: NVIDIA H/B-series, AMD MI-series, Google TPU, Intel Gaudi.
Why It Matters in AI
LLM inference and training are memory-bandwidth-bound: the GPU spends more time waiting for weights and activations than on arithmetic. HBM3e on the H200 delivers 4.8 TB/s versus ~1 TB/s for GDDR6X — a 4–5× bandwidth advantage that translates directly into faster matrix multiplications and larger effective batch sizes. HBM is manufactured almost exclusively by SK Hynix, Samsung, and Micron, making it a geopolitical chokepoint: export controls on HBM to China are a central tool in US chip policy.
Key Points
| Aspect | Description |
|---|---|
| Capacity | Up to 192 GB per GPU (AMD MI300X, NVIDIA B200) — fits 70B model weights without offloading |
| Stacking | 4–12 DRAM dies bonded vertically via TSVs — stacks sit millimetres from GPU die on interposer |
| Bus width | 1,024 bits per stack (HBM2e), 1,024 bits (HBM3/3e) — vs 384-bit for GDDR6X |
| Suppliers | SK Hynix (~50% market share), Samsung, Micron — HBM3e supply constrained NVIDIA in 2024 |
| Generations | HBM2 → HBM2e → HBM3 → HBM3e — bandwidth roughly doubles each generation |
| H100 vs H200 | H100: 80 GB HBM3, 3.35 TB/s; H200: 141 GB HBM3e, 4.8 TB/s — same die, memory upgrade only |
Simple Analogy
A highway versus a country lane: GDDR is a fast single-lane road — high speed but limited throughput. HBM is a thousand-lane motorway running in parallel — each lane slower than GDDR, but the aggregate throughput dwarfs anything a narrow bus can achieve. Stacking DRAM dies is how you build a thousand lanes in the space of a postage stamp.
Common Usage Examples
- NVIDIA H100 SXM: 6 × HBM3 stacks, 80 GB total, 3.35 TB/s — connected via CoWoS interposer
- NVIDIA H200: 6 × HBM3e stacks, 141 GB, 4.8 TB/s — drop-in upgrade for H100 servers
- AMD MI300X: 8 × HBM3 stacks, 192 GB, 5.3 TB/s — largest HBM capacity per GPU as of 2024
- Google TPU v5p: HBM2e, 95 GB per chip — powers Gemini training at scale
nvidia-smi --query-gpu=memory.total --format=csv— reports HBM capacity on NVIDIA GPUs
Summary
In short: HBM stacks DRAM dies vertically and connects them via a massive parallel bus, delivering the terabytes-per-second memory bandwidth that AI accelerators need — and its constrained supply from three Korean/US manufacturers makes it a critical bottleneck for global AI hardware production.