← AI Terminology
LLMOps
LLMOps is the operational discipline for large language model applications: prompt/version control, eval harnesses, tracing, cost control, safety, and deployment of LLM systems.
It extends MLOps with LLM-specific failure modes.
It extends MLOps with LLM-specific failure modes.
Why It Matters in AI
LLM apps break via prompts, tools, and non-determinism, not only data drift. LLMOps adds eval sets, LLM-as-judge, guardrails, and token budgets to classical MLOps. Teams that skip it cannot ship reliably.
Key Points
| Aspect | Description |
|---|---|
| Scope | Prompts, chains, RAG, agents, models, costs |
| Tools | LangSmith, Phoenix, Langfuse, Helicone, PromptLayer |
| Safety | Injection tests, PII, policy regression |
| Related | MLOps, observability, golden datasets |
| Vs MLOps | More text evals; less classical feature stores |
| Practices | Version prompts, golden sets, CI evals, tracing |
Simple Analogy
Airline ops for a new kind of aircraft: still dispatch and maintenance, but new checklists for software that writes and tools that act.
Common Usage Examples
- CI runs eval suite on prompt PRs
- Trace every tool call in prod
- Budget alerts on tokens/$
- Regression tests for jailbreaks
Summary
In short: LLMOps operates LLM products end-to-end — evals, tracing, cost, and safety beyond classical MLOps.