← AI Terminology
Hyperparameter
A hyperparameter is a configuration value that controls the training process itself — as opposed to a parameter (model weight), which is learned from data.
Hyperparameters are set before training begins and determine how the model learns, not what it learns.
Hyperparameters are set before training begins and determine how the model learns, not what it learns.
Why It Matters in AI
The right learning rate can mean the difference between a model that converges in hours and one that never converges. Hyperparameters — learning rate, batch size, number of layers, dropout rate, weight decay — control every aspect of the training dynamics. Finding good hyperparameters for a new model or dataset often requires systematic search (grid search, Bayesian optimisation) and deep understanding of how each one affects training.
Key Points
| Aspect | Description |
|---|---|
| Tuning | Grid search, random search, Bayesian optimisation, population-based training (PBT) |
| Batch size | Affects gradient noise and effective learning rate; larger requires proportionally larger LR |
| Training HP | Epochs, warmup steps, LR schedule, gradient clip threshold |
| Learning rate | Single most important hyperparameter — too high: diverges; too low: never converges |
| Architecture HP | Number of layers, hidden units, attention heads — define model capacity |
| Regularisation HP | Dropout rate, weight decay (L2), L1 coefficient — control overfitting |
Simple Analogy
Baking: the recipe (model architecture) and oven settings (hyperparameters) are distinct. The recipe lists ingredients (weights/parameters) that change as the dough rises (training). The oven temperature (learning rate) and bake time (epochs) are set before baking — change them and the same recipe produces a completely different result.
Common Usage Examples
--learning_rate 2e-5 --num_train_epochs 3 --per_device_train_batch_size 16— HuggingFace Trainer- Grid search:
param_grid = {'lr': [0.001, 0.01], 'dropout': [0.1, 0.3]}; GridSearchCV(model, param_grid) - Optuna:
lr = trial.suggest_float('lr', 1e-5, 1e-2, log=True)— Bayesian hyperparameter optimisation - Learning rate finder: use
lr_find()in PyTorch Lightning before starting training - Wandb sweep:
wandb agent sweep_id— distributed HPO with experiment tracking
Summary
In short: Hyperparameters control how training happens — unlike weights, which are learned from data, hyperparameters are set by the practitioner and profoundly affect whether and how well the model learns.