← AI Terminology
Knowledge Distillation
Knowledge distillation is a model compression technique where a smaller "student" model is trained to mimic the output distribution of a larger, more capable "teacher" model — transferring the teacher's knowledge into a more efficient form.
The student learns from soft probability outputs (not just hard labels), capturing richer information about the teacher's learned representations.
The student learns from soft probability outputs (not just hard labels), capturing richer information about the teacher's learned representations.
Why It Matters in AI
Distillation makes large models deployable: a GPT-4-class teacher can train a GPT-3.5-class student that runs 10× faster at similar quality. It is how OpenAI produces cheaper API tiers, how mobile models (DistilBERT, MobileNet) match heavy models, and how many edge AI deployments achieve frontier-model quality. It is also central to model proliferation risk: frontier model capabilities can be distilled into open-source models that bypass safety training.
Key Points
| Aspect | Description |
|---|---|
| DistilBERT | 6-layer student of 12-layer BERT — 40% smaller, 60% faster, 97% of BERT performance |
| Soft labels | Teacher outputs probability distributions (e.g. 70% cat, 25% dog) — richer than hard labels |
| Loss function | KL-divergence between student and teacher distributions, added to student's task loss |
| Temperature T | Higher T (>1) softens distributions, exposing more inter-class similarity for the student |
| Proliferation risk | Distilling GPT-4 outputs into LLaMA creates capable open models outside safety controls |
| Feature distillation | Match intermediate layer activations, not just output logits — "FitNets" approach |
Simple Analogy
A senior expert teaching a junior: instead of just giving yes/no answers, the expert explains their reasoning with degrees of certainty ("probably A, but B is plausible"). The junior learns not just the answer but the expert's nuanced way of thinking — far more than they'd get from textbook labels alone.
Common Usage Examples
- DistilBERT:
from transformers import DistilBertModel— 6-layer BERT distilled by HuggingFace loss = alpha * task_loss + (1-alpha) * kl_div(student_logits/T, teacher_logits/T)- TinyBERT, MobileBERT: distilled BERT variants for mobile/edge deployment
trl.GKDTrainer— Generalized KD trainer in HuggingFace TRL for LLM distillation- OpenAI usage policy: prohibits using ChatGPT outputs to train competing models (distillation risk)
Summary
In short: Knowledge distillation compresses a large model's capabilities into a smaller, faster student by training on the teacher's probability outputs — the primary technique for making large-model quality accessible at inference scale.