← AI Terminology

Self-Supervised Learning

Self-supervised learning is a training paradigm where labels are automatically generated from the input data itself — no human annotation required — allowing models to learn rich representations from massive unlabelled datasets.

Pre-training LLMs on next-token prediction is the most prominent example.
Why It Matters in AI
Human-labelled datasets are expensive and limited: ImageNet took years and millions of dollars to annotate. The internet contains trillions of unlabelled tokens, images, and videos. Self-supervised learning exploits this: by predicting masked tokens (BERT), the next token (GPT), a missing image patch (MAE), or matching augmented views (contrastive learning), models learn representations that transfer to downstream tasks with very few labels. This is the training paradigm behind all LLMs and modern vision foundation models.
Key Points
Aspect Description
Causal LM Predict the next token from all prior tokens — GPT, LLaMA; autoregressive language modelling
Contrastive Maximise similarity of augmented views of the same image; minimise similarity of different images — SimCLR, MoCo
Foundation model Self-supervised pre-training at scale → generally capable model — GPT-4, CLIP, SAM
Masked autoencoder Mask image patches; reconstruct — MAE, BEiT; learns visual representations without labels
Downstream transfer Pre-trained representation fine-tuned on a small labelled dataset — labels needed only for final task
Masked language model Mask 15% of tokens; predict them — BERT, RoBERTa; learns contextual representations
Simple Analogy
A child learning language by listening: no teacher labels every word — the child learns from the patterns in the stream of language, predicting what comes next, learning that words in similar contexts have similar meanings. Self-supervised learning is this natural learning process, applied to neural networks at massive scale.
Common Usage Examples
  • GPT pre-training: loss = cross_entropy(model(input_ids), shifted_input_ids) — next-token prediction
  • BERT: MLMHead predicts masked tokens; NSPHead predicts next sentence — both self-supervised
  • MAE: ViTMAE.from_pretrained("facebook/vit-mae-large") — masked image reconstruction pre-training
  • SimCLR: loss = NTXentLoss(); loss(z_i, z_j) — contrastive loss on augmented image pairs
  • DINO (Meta): self-distillation from a momentum teacher — self-supervised ViT for vision tasks
Summary
In short: Self-supervised learning generates labels automatically from data itself — enabling pre-training on internet-scale unlabelled corpora, the paradigm underlying all LLMs and modern vision foundation models.