← AI Terminology
SFT - Supervised Fine-Tuning
SFT (Supervised Fine-Tuning) is the training phase that converts a raw pre-trained language model into an instruction-following assistant — by fine-tuning on a curated dataset of (instruction, response) pairs using standard cross-entropy loss on the response tokens.
It is the first step in the RLHF pipeline, before reward modelling and PPO.
It is the first step in the RLHF pipeline, before reward modelling and PPO.
Why It Matters in AI
A pre-trained LLM continues text; it doesn't respond to instructions. SFT teaches the model that instructions are commands to execute — not text to complete. It is the lowest-cost, highest-impact alignment step: a model trained on 1,000–100,000 high-quality instruction-response pairs learns to follow instructions, adopt a helpful persona, and format outputs appropriately. InstructGPT showed that a 1.3B SFT model was preferred by humans over a 175B base GPT-3 — quality of SFT data matters more than scale.
Key Points
| Aspect | Description |
|---|---|
| Loss | Cross-entropy on response tokens only — instruction tokens are masked (not learned from) |
| LoRA SFT | Fine-tune via LoRA adapters — common for adapting open-source models on consumer hardware |
| After SFT | Usually followed by RLHF/DPO to improve alignment, safety, and preference satisfaction |
| Training data | (instruction, [input,] response) triples — Alpaca, ShareGPT, Dolly, FLAN formats |
| Dataset quality | Quality >> quantity — 10K high-quality examples often outperforms 1M noisy ones |
| vs Instruction tuning | Same concept; SFT is the technical ML name; "instruction tuning" is more descriptive |
Simple Analogy
Training an intern: they know everything from years of reading (pre-training) but haven't learned professional norms — how to respond to requests, structure output, and be helpful. SFT is the first week of onboarding where they learn that emails end with "Best regards," tasks get prioritised, and questions deserve direct answers.
Common Usage Examples
from trl import SFTTrainer; trainer = SFTTrainer(model, train_dataset=alpaca_dataset, peft_config=lora_config)- Alpaca format:
{"instruction": "...", "input": "...", "output": "..."}— standard SFT schema formatting_func = lambda x: f"### Instruction:\n{x['instruction']}\n\n### Response:\n{x['output']}"— chat templatetrainer.train()— runs SFT on formatted instruction-response pairs- Databricks Dolly: 15K instruction examples used to SFT Pythia-12B into Dolly — open instruction model
Summary
In short: SFT converts a raw pre-trained LLM into an instruction-following assistant by training on curated (instruction, response) pairs — the essential first alignment step, where data quality matters more than quantity.