← AI Terminology
Computer Use
Computer use is an agent capability where a model operates a computer GUI or browser — seeing screenshots or accessibility trees and issuing clicks, typing, and navigation actions.
It extends tool use from APIs to the full desktop/web environment.
It extends tool use from APIs to the full desktop/web environment.
Why It Matters in AI
Most real work lives in UIs without clean APIs. Computer-use agents automate legacy software, web workflows, and QA. They raise new reliability and safety stakes (irreversible clicks, credentials, prompt injection via page content).
Key Points
| Aspect | Description |
|---|---|
| I/O | Screenshots/DOM in; mouse/keyboard/actions out |
| Eval | WebArena, OSWorld-like desktop tasks |
| Risks | Prompt injection from web, data exfil, destructive actions |
| Stack | Playwright/Selenium + VLM perception + planner |
| Related | Browser use, RPA, tool use |
| Products | Anthropic computer use, OpenAI Operator-style agents, browser agents |
Simple Analogy
A remote intern who watches your screen over video and moves the mouse for you — powerful, but you still need guardrails on what they can click.
Common Usage Examples
- Anthropic computer-use tool demos
- Browser agents booking flows
- Desktop RPA with vision-language models
- Sandbox VMs for safe execution
Summary
In short: Computer use lets agents control GUIs and browsers with vision and actions — automating software that has no clean API.