← AI Terminology
VLM - Vision-Language Model
A vision-language model (VLM) jointly processes images (and sometimes video) with text — enabling captioning, visual Q&A, screenshot agents, and multimodal chat.
Examples include GPT-4V-class systems, Gemini, LLaVA, and open VLMs.
Examples include GPT-4V-class systems, Gemini, LLaVA, and open VLMs.
Why It Matters in AI
The world is not text-only. VLMs ground language in pixels for UI agents, document understanding, and accessibility. They are central to multimodal product roadmaps.
Key Points
| Aspect | Description |
|---|---|
| Open | LLaVA, Qwen-VL, Idefics-class models |
| Risks | Visual hallucinations; prompt injection in images |
| Tasks | Captioning, VQA, OCR-rich QA, GUI control |
| Related | CLIP, multimodal, computer use |
| Training | Image-text pairs, instruction tuning, preference data |
| Architecture | Vision encoder + LLM + projector/cross-attn fusion |
Simple Analogy
A colleague who can look at a slide and discuss it — eyes and language in one brain.
Common Usage Examples
- LLaVA-style open VLMs
- GPT-4o/Claude/Gemini image inputs
- Screenshot-to-action agents
- Eval on VQAv2 / TextVQA / UI benches
Summary
In short: Vision-language models combine sight and text in one system — the foundation of multimodal chat and visual agents.