← AI Terminology
NPU - Neural Processing Unit
An NPU (Neural Processing Unit) is a dedicated hardware accelerator designed specifically for neural network inference — optimised for the matrix multiplications, convolutions, and activation functions that dominate AI compute, with far lower power consumption than GPUs for on-device inference.
Found in every modern smartphone (Apple A-series, Qualcomm Snapdragon, Samsung Exynos) and AI PC chips.
Found in every modern smartphone (Apple A-series, Qualcomm Snapdragon, Samsung Exynos) and AI PC chips.
Why It Matters in AI
GPUs excel at training and cloud inference but are power-hungry — unsuitable for battery-powered devices. NPUs solve this: a dedicated silicon block that runs quantised neural network models at milliwatts rather than hundreds of watts, enabling real-time AI on phones, laptops, and embedded devices without cloud round-trips. Apple Neural Engine processes face detection and FaceID; Qualcomm Hexagon runs on-device LLMs. As on-device AI grows, NPUs are the critical enabling hardware.
Key Points
| Aspect | Description |
|---|---|
| Apple ANE | Apple Neural Engine in A17/M3: up to 35 TOPS — runs Core ML models (FaceID, Siri, Photos AI) |
| Intel NPU | Meteor Lake / Lunar Lake: dedicated NPU for Windows AI PC — runs ONNX models via DirectML |
| Precision | INT8/INT4 quantised inference — neural networks tolerate low precision, enabling tiny silicon |
| Architecture | Systolic arrays or vector engines for matrix multiply — fixed-function or programmable |
| Power efficiency | ~10–100× more efficient than GPU for inference — measured in TOPS/W (tera-ops per watt) |
| Qualcomm Hexagon | Snapdragon NPU: up to 75 TOPS on Snapdragon 8 Gen 3 — runs on-device LLMs (Llama, Mistral) |
Simple Analogy
A specialised assembly line for one product vs. a general-purpose factory floor: a GPU is the versatile factory that can make anything but uses enormous energy; an NPU is the purpose-built line that makes neural network inferences at a fraction of the cost, optimised for that single task.
Common Usage Examples
- Apple Core ML:
MLModel(contentsOf: modelURL)— automatically routes to ANE for supported operations - Qualcomm AI Engine SDK: deploy ONNX/TFLite models to Hexagon NPU via
qnn-net-run - ONNX Runtime on Windows:
ort.InferenceSession(model, providers=["DmlExecutionProvider"])— Intel NPU torch.backends.nnapi.is_available()— check Android NPU availability from PyTorch Mobile- MediaPipe on Android: face mesh, hand tracking — runs on mobile NPU for real-time <5ms inference
Summary
In short: An NPU is a purpose-built chip for neural network inference — delivering GPU-class AI performance at a fraction of the power, enabling real-time on-device AI on smartphones and laptops.