← AI Terminology
Multimodal
Multimodal refers to AI systems that process and reason over multiple types of data — such as text, images, audio, video, and code — within a single model, rather than using separate specialist models for each modality.
Modern frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5) are all multimodal.
Modern frontier models (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5) are all multimodal.
Why It Matters in AI
Real-world intelligence is inherently multimodal: humans read, see, hear, and speak simultaneously. AI models confined to text cannot analyse a medical scan, interpret a chart, transcribe speech, or reason about a video. Multimodal capabilities unlock the most valuable real-world use cases — document understanding, visual QA, scientific image analysis, and voice interaction — and represent the frontier of AI product development.
Key Points
| Aspect | Description |
|---|---|
| Any-to-any | Any modality in, any out (GPT-4o, Gemini 1.5 Flash) — text, image, audio, video |
| Applications | Medical imaging + report generation, chart QA, video summarisation, voice assistants |
| Key benchmarks | MMMU (multi-discipline multimodal), DocVQA (document QA), MMBench (multimodal evaluation) |
| Vision encoder | Separate CNN/ViT encodes images into token embeddings fed into the LLM — e.g. CLIP + LLM |
| Vision-language | Image + text input; text output — most common multimodal capability (GPT-4V, Claude Vision) |
| Native multimodal | End-to-end training on all modalities simultaneously — better cross-modal reasoning |
Simple Analogy
A person who can not only read a report but also look at the graphs in it, listen to the audio briefing, and watch the demo video — all as part of the same understanding. A multimodal model does the same: it doesn't just read the text of a document, it sees the figures and tables embedded in it.
Common Usage Examples
- Claude vision:
[{"type": "image", "source": {"type": "base64", "data": "..."}}, {"type": "text", "text": "Describe this chart"}] - OpenAI GPT-4o:
messages=[{"role": "user", "content": [{"type": "image_url", "image_url": ...}, {"type": "text", "text": "What's in this image?"}]}] - Whisper: speech → text (audio-to-text modality)
- CLIP: joint image-text embedding space — foundation of many multimodal systems
- Gemini 1.5: native video understanding — analyse a 1-hour video with a single API call
Summary
In short: Multimodal AI processes text, images, audio, and video in a single model — reflecting how intelligence actually works and unlocking the highest-value real-world applications that text-only models cannot address.