← AI Terminology
Validation Set
A validation set is a held-out subset of labelled data — separate from the training set — used to evaluate model performance during training to guide hyperparameter tuning and early stopping without contaminating the final test set.
It is the feedback loop that prevents overfitting to the training data.
It is the feedback loop that prevents overfitting to the training data.
Why It Matters in AI
Training loss always decreases with more training — it measures memorisation, not generalisation. The validation set provides an honest signal during training: when validation loss stops improving, the model is beginning to overfit. Without it, every hyperparameter choice (learning rate, architecture, regularisation) would require a final test-set evaluation, burning the only unbiased estimate of true performance.
Key Points
| Aspect | Description |
|---|---|
| Split | Typically 10–20% of labelled data; remainder split ~80/10/10 train/val/test |
| Dev set | Machine learning literature term for validation set — same concept, different name |
| Data leakage | Preprocessing (scaling, mean) must be fit on train only, then applied to val and test |
| Early stopping | Stop training when val loss has not improved for N epochs — prevents overfitting |
| Hyperparameter | Choose LR, depth, dropout, regularisation by watching val loss/metric |
| Cross-validation | K-fold: rotate which fold is validation — better estimate with limited data |
Simple Analogy
Practice exam questions from a past year: you study from your notes (training), then test yourself on last year's paper (validation) to adjust your approach. The actual exam (test set) is only seen once — using it repeatedly to guide your study would inflate your confidence and undermine the point of the exam.
Common Usage Examples
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42)- PyTorch:
DataLoader(val_dataset, batch_size=32, shuffle=False)— no shuffling for reproducible eval - Keras:
model.fit(X_train, y_train, validation_data=(X_val, y_val), callbacks=[EarlyStopping()]) GridSearchCV(cv=5)— 5-fold cross-validation as validation strategy without a fixed val split- Monitor
val_lossin TensorBoard/W&B to detect overfitting during training runs
Summary
In short: The validation set is training's honest mirror — the held-out data that guides hyperparameter choices and early stopping without spending the test set, keeping the final evaluation unbiased.