← AI Terminology
llama.cpp
llama.cpp is a popular C/C++ inference engine for running LLaMA-family and many other LLMs efficiently on CPUs and GPUs, centered on GGUF models.
It democratised local LLMs on consumer hardware.
It democratised local LLMs on consumer hardware.
Why It Matters in AI
Not everyone has datacenter GPUs. llama.cpp’s quantisation and tight kernels made laptop LLMs practical and spawned Ollama/LM Studio. It is the reference for edge LLM engineering.
Key Points
| Aspect | Description |
|---|---|
| Dev | Active open-source community |
| Role | Core engine under many chat UIs |
| Origin | Georgi Gerganov et al.; ggml ecosystem |
| Related | GGUF, Ollama, whisper.cpp |
| Backends | CPU SIMD, Metal, CUDA, Vulkan, etc. |
| Features | GGUF, grammars, embeddings, server mode |
Simple Analogy
A lightweight race engine that fits in a garage and still drives surprisingly far — local, hackable, efficient.
Common Usage Examples
./llama-cli -m model.gguf -p "..."- llama-server OpenAI-compatible API
- Metal accelerate on Mac
- Grammar-constrained JSON locally
Summary
In short: llama.cpp is the efficient C++ engine that made quantized local LLMs practical on everyday hardware.