← AI Terminology
On-Device AI
On-device AI runs models locally on phones, laptops, or embedded hardware rather than sending data to cloud GPUs.
Enabled by NPUs, quantisation, and efficient architectures.
Enabled by NPUs, quantisation, and efficient architectures.
Why It Matters in AI
Privacy, offline UX, and marginal cost drive on-device inference. Apple, Qualcomm, and PC NPUs now ship capable SLMs and vision models. Product design splits what stays local vs cloud.
Key Points
| Aspect | Description |
|---|---|
| Limits | Model size, thermals, battery |
| Stacks | Core ML, MLX, ExecuTorch, ONNX Runtime Mobile |
| Related | SLM, edge AI, GGUF |
| Benefits | Privacy, offline, low latency, no per-token cloud bill |
| Hardware | NPUs, Apple Silicon, Snapdragon, NPUs in PCs |
| Techniques | Quantisation, distillation, small multimodal models |
Simple Analogy
Cooking at home instead of ordering delivery every meal — more private and often faster, with a smaller kitchen.
Common Usage Examples
- iOS on-device summarisation features
- llama.cpp on a laptop CPU/GPU
- Camera models running fully offline
- Hybrid: local draft + cloud escalate
Summary
In short: On-device AI runs models locally on user hardware — privacy-friendly, offline-capable intelligence at the edge.