← AI Terminology
vLLM
vLLM is an open-source LLM inference engine optimised for high-throughput and memory-efficient serving — achieving up to 24× higher throughput than HuggingFace Transformers by implementing PagedAttention for efficient KV cache management.
Developed at UC Berkeley (2023); now the de facto standard for production LLM serving.
Developed at UC Berkeley (2023); now the de facto standard for production LLM serving.
Why It Matters in AI
LLM inference is bottlenecked by KV cache memory: each request needs contiguous GPU memory for its attention keys/values, leading to fragmentation and waste. vLLM's PagedAttention manages KV cache in non-contiguous pages (like OS virtual memory), enabling continuous batching across requests of varying lengths and dramatically increasing GPU utilisation. A single vLLM server can serve hundreds of concurrent users that would otherwise require separate inference processes.
Key Points
| Aspect | Description |
|---|---|
| Quantization | AWQ, GPTQ, FP8, INT8 support — reduced VRAM with minimal accuracy loss |
| Model support | Llama, Mistral, Qwen, Gemma, Phi, DeepSeek, Falcon, and 50+ architectures via HuggingFace |
| PagedAttention | KV cache stored in fixed-size pages, mapped via a block table — eliminates fragmentation |
| OpenAI-compatible | Serves /v1/completions and /v1/chat/completions — drop-in replacement for OpenAI API |
| Tensor parallelism | --tensor-parallel-size N — shard model across N GPUs for models exceeding one GPU |
| Continuous batching | New requests join mid-batch as soon as a slot opens — maximises GPU utilisation |
Simple Analogy
A restaurant with an open kitchen versus one where each waiter stands at a fixed stove: HuggingFace inference reserves a stove (GPU memory) per order even between courses. vLLM runs a shared kitchen where burners are assigned only while actively cooking, and the moment one dish is plated, that burner serves the next order — the same kitchen handles 20× more covers.
Common Usage Examples
pip install vllm && vllm serve meta-llama/Llama-3-8B-Instruct --tensor-parallel-size 1- Python API:
LLM(model="mistralai/Mistral-7B-Instruct-v0.3").generate(prompts) - OpenAI client:
OpenAI(base_url="http://localhost:8000/v1", api_key="token").chat.completions.create(...) --quantization awq— load AWQ-quantised model, halves VRAM vs fp16- Docker:
docker run --gpus all vllm/vllm-openai:latest --model <hf-model>
Summary
In short: vLLM is the high-throughput LLM serving engine that solved KV cache memory fragmentation with PagedAttention — enabling continuous batching and 24× throughput gains, making it the standard for production inference at scale.