← AI Terminology
ViT - Vision Transformer
ViT (Vision Transformer) is an image classification architecture that splits an image into fixed-size patches, treats each patch as a token, and processes them with a standard transformer encoder — demonstrating that attention alone, without convolutions, achieves state-of-the-art vision performance at scale.
Introduced by Dosovitskiy et al. (Google Brain, 2020); it unified vision and language architecture.
Introduced by Dosovitskiy et al. (Google Brain, 2020); it unified vision and language architecture.
Why It Matters in AI
CNNs dominated vision for a decade, but their inductive biases (local receptive fields, translation invariance) cap scalability. ViT showed that transformers scale better than CNNs on large datasets — with enough data and compute, a pure attention model surpasses ResNets. This opened the door to multimodal models: the same architecture now handles image, text, video, and audio. ViT variants (DeiT, CLIP, DINO, SAM, SigLIP) underpin most modern vision and multimodal systems.
Key Points
| Aspect | Description |
|---|---|
| Scale | ViT-B (86M params), ViT-L (307M), ViT-H (632M) — larger models close the gap with CNNs |
| Variants | DeiT (data-efficient), Swin (hierarchical windows), DINO (self-supervised), SAM (segmentation) |
| Multimodal | CLIP, ALIGN, SigLIP use ViT as image encoder paired with text transformer |
| Pre-training | Requires large datasets (JFT-300M, ImageNet-21k) — less sample-efficient than CNNs at small scale |
| Positional enc | Learnable 1D positional embeddings added to patch tokens — no 2D inductive bias |
| Patch embedding | 224×224 image → 196 patches of 16×16 px → linear projection → 196 tokens + [CLS] token |
Simple Analogy
Reading an image like a document: instead of scanning pixels with a sliding window (CNN), slice the image into a grid of tiles and read the tiles as words in a sentence. Attention lets every tile "look at" every other tile simultaneously — a painting's corner can directly influence how the centre is interpreted, just as words in a sentence inform each other.
Common Usage Examples
from timm import create_model; vit = create_model("vit_base_patch16_224", pretrained=True)- HuggingFace:
ViTForImageClassification.from_pretrained("google/vit-base-patch16-224") - Fine-tune:
model.heads.head = nn.Linear(768, num_classes)— replace classification head - DINO:
ViTFeatureExtractor— self-supervised patch features for downstream tasks - SAM (Segment Anything): ViT-H as image encoder — produces dense embeddings for mask prediction
Summary
In short: ViT proved that transformers applied to image patches match and exceed CNNs at scale — unifying vision and language under one architecture and becoming the image encoder of choice for multimodal AI systems.