← AI Terminology
Transfer Learning
Transfer learning is the practice of taking a model pre-trained on a large dataset for a general task and adapting it to a specific downstream task — either by fine-tuning some or all layers with a smaller task-specific dataset, or by using the pre-trained model as a feature extractor.
It is the dominant paradigm in modern deep learning: pre-train once, fine-tune many times.
It is the dominant paradigm in modern deep learning: pre-train once, fine-tune many times.
Why It Matters in AI
Training a large model from scratch on ImageNet requires weeks on hundreds of GPUs. Transfer learning lets every practitioner benefit from that investment: download the pre-trained ResNet-50, fine-tune the final layer on 1,000 custom images in hours on one GPU. This pattern — large pre-training on general data → small fine-tuning on specific data — underlies every LLM deployment (GPT-4 → ChatGPT → fine-tuned chatbots), every vision system (ImageNet → medical imaging), and every NLP tool (BERT → sentiment classifier).
Key Points
| Aspect | Description |
|---|---|
| Few-shot | Transfer from large pre-training enables good performance with very few target examples |
| Fine-tuning | Update some or all pre-trained layers with a low learning rate — better accuracy, more compute |
| Domain shift | If target domain differs from pre-training domain, fine-tune deeper layers to adapt |
| Foundation models | LLMs pre-trained on internet text → fine-tuned for specific tasks — transfer at scale |
| Negative transfer | If domains are very different, pre-training can hurt — random init may be better (rare) |
| Feature extraction | Freeze all pre-trained layers; train only a new output head — fast, few trainable parameters |
Simple Analogy
A chef who trained for 10 years in French cuisine (pre-training on large, general data) switching to a Japanese restaurant: they don't start from zero — knife skills, flavour balancing, and food science transfer directly. They need a few weeks learning Japanese-specific techniques (fine-tuning), not 10 more years.
Common Usage Examples
model = torchvision.models.resnet50(pretrained=True); model.fc = nn.Linear(2048, num_classes)— feature extractionfor param in model.parameters(): param.requires_grad = False; model.fc.requires_grad = True— freeze backbone- Fine-tuning:
optimizer = Adam([{'params': model.fc.parameters(), 'lr': 1e-3}, {'params': model.layer4.parameters(), 'lr': 1e-4}]) - BERT fine-tune:
AutoModelForSequenceClassification.from_pretrained("bert-base-uncased", num_labels=2) - LoRA fine-tuning:
get_peft_model(llama3, LoraConfig(...))— parameter-efficient transfer for LLMs
Summary
In short: Transfer learning adapts a large pre-trained model to a specific task with minimal additional data and compute — the paradigm underlying all modern AI deployment, where pre-training costs are amortised across thousands of downstream applications.