← AI Terminology
GGUF
GGUF is a binary file format for quantized ML models (successor spirit to GGML formats) widely used by llama.cpp and the local-LLM ecosystem.
It packages tensors, metadata, and quant types for portable inference.
It packages tensors, metadata, and quant types for portable inference.
Why It Matters in AI
Local AI exploded via single-file GGUF models shared on Hugging Face. Understanding GGUF quant levels (Q4_K_M, Q5_K_S, …) is required to run LLMs on laptops and phones via llama.cpp/Ollama.
Key Points
| Aspect | Description |
|---|---|
| Pro | Portable, simple distribution |
| Quants | Many GGML-family quant schemes in one format |
| Convert | HF → GGUF conversion scripts |
| Related | llama.cpp, AWQ/GPTQ (different stacks) |
| Contains | Weights, tokenizer info, architecture metadata |
| Ecosystem | llama.cpp, Ollama, LM Studio, GPT4All |
Simple Analogy
A USB stick format everyone agreed on for pocket-sized brains — plug into many apps without re-encoding chaos.
Common Usage Examples
- Download
model-Q4_K_M.gguf llama.cppload GGUF- Ollama Modelfile FROM gguf
- Pick quant by VRAM/RAM budget
Summary
In short: GGUF is the portable quantized model file format powering most local LLM tools built around llama.cpp.