← AI Terminology
Prompt Caching
Prompt caching reuses computation (and often offers cheaper pricing) for long shared prefixes across requests — system prompts, large docs, or static tool schemas.
Providers and engines cache KV states or billed prefix tokens.
Providers and engines cache KV states or billed prefix tokens.
Why It Matters in AI
Agent and RAG apps resend huge stable prefixes every call. Caching cuts cost and TTFT dramatically when the beginning of the prompt repeats. Designing prompts for cache-friendly stable prefixes is now an economic skill.
Key Points
| Aspect | Description |
|---|---|
| Design | Put static content first; dynamic user text last |
| Benefit | Lower $ and faster prefill |
| Related | Prefix caching, continuous batching |
| Providers | Anthropic, OpenAI, Gemini caching products; vLLM prefix caching |
| What caches | KV for shared token prefix; provider-side bill credits |
| Invalidation | Any prefix change misses cache |
Simple Analogy
A coffee shop pre-grinding the house blend every morning — shared prep work reused for every customer who orders the same base drink.
Common Usage Examples
- Anthropic
cache_controlephemeral breakpoints - OpenAI prompt caching on long prefixes
- vLLM automatic prefix caching
- Stable system prompt + tools before user message
Summary
In short: Prompt caching reuses work on repeated prompt prefixes — slashing cost and prefill latency for apps with large stable context.