← AI Terminology
Contrastive Learning
Contrastive learning is a self-supervised learning approach where a model is trained to bring representations of similar (positive) pairs close together and push representations of dissimilar (negative) pairs apart in an embedding space.
It learns useful representations from unlabelled data by defining similarity through data augmentation.
It learns useful representations from unlabelled data by defining similarity through data augmentation.
Why It Matters in AI
Contrastive learning enabled the visual pre-training revolution: SimCLR, MoCo, and BYOL showed that self-supervised models could match or exceed supervised ImageNet pre-training without labels. CLIP extended the idea to image-text pairs. It is now the dominant paradigm for learning general-purpose embeddings across vision, NLP, audio, and multimodal domains.
Key Points
| Aspect | Description |
|---|---|
| CLIP | Contrastive learning between image and text encoders on 400M web pairs |
| SimCLR | Simple framework: two augmentations, projection head, InfoNCE — first to match supervised perf |
| InfoNCE loss | Contrastive loss function: maximise similarity of positives vs. negatives in a batch |
| BYOL / SimSiam | Bootstrap-only: no negatives needed — uses a momentum encoder to avoid collapse |
| Negative pairs | Different data points — representations should be pushed far apart |
| Positive pairs | Two augmented views of the same data point — they should have similar representations |
Simple Analogy
Training a dog to tell apart dogs from cats by showing it many pairs: "these two are both dogs — learn what makes them similar; this dog and this cat are different — learn what separates them." The model doesn't need labels; the pairing signal is enough to learn the concept.
Common Usage Examples
- SimCLR: two random crops of the same image are a positive pair; all others in the batch are negatives
torch.nn.CosineEmbeddingLoss— primitive contrastive loss for embedding alignmentsentence-transformerslibrary: contrastively trained text encoders for semantic search- CLIP: image + its caption = positive; image + random caption = negative
- NT-Xent loss (SimCLR):
L = -log(exp(sim(z_i, z_j)/τ) / Σ exp(sim(z_i, z_k)/τ))
Summary
In short: Contrastive learning teaches a model what things are by showing it what they are similar to and different from — enabling powerful representations without any labels.