← AI Terminology
Bayesian Optimization
Bayesian optimization is a sequential strategy for optimising expensive-to-evaluate black-box functions by building a probabilistic surrogate model (usually a Gaussian process) and using it to decide where to sample next.
It is the gold standard for hyperparameter tuning when each evaluation (a training run) costs minutes or hours.
It is the gold standard for hyperparameter tuning when each evaluation (a training run) costs minutes or hours.
Why It Matters in AI
Grid search and random search waste evaluations on uninformative regions. Bayesian optimization learns from past trials to predict which hyperparameter combinations are most likely to improve the objective, then explicitly balances exploration (try uncertain regions) and exploitation (refine promising ones). It typically finds better hyperparameters in far fewer trials — critical when each trial costs cloud GPU hours.
Key Points
| Aspect | Description |
|---|---|
| Tools | Optuna, Hyperopt, Ray Tune, BoTorch (PyTorch), W&B Sweeps, SigOpt |
| Contrast | Grid search: O(n^d) evaluations. Bayesian: typically 50–200 trials regardless of dimension |
| Scalability | Gaussian processes scale poorly beyond ~20 dims; TPE (Optuna) or SMAC used for larger spaces |
| Surrogate model | Gaussian process (or Random Forest/neural net) — models P(objective |
| Exploration/exploit | Acquisition function explicitly trades off trying new regions vs. refining known good ones |
| Acquisition function | Selects next point to evaluate — Expected Improvement (EI), UCB, or Thompson sampling |
Simple Analogy
Drilling for oil: a naive approach tries random locations; a grid search tries evenly-spaced spots; Bayesian optimisation acts like a geologist who analyses each dry well to update a probability map of where oil is likely, then picks the next spot that maximises expected yield given what's been learned.
Common Usage Examples
study = optuna.create_study(direction='maximize'); study.optimize(objective, n_trials=100)- W&B Sweeps with
method: bayes— integrates Bayesian HPO with experiment tracking BayesSearchCVfromscikit-optimize— drop-in replacement for scikit-learn'sGridSearchCV- BoTorch:
optimize_acqf(EI, bounds=bounds)for advanced acquisition function optimisation - Hyperparameter tuning GPT fine-tuning runs: Bayesian search over lr, warmup, batch size
Summary
In short: Bayesian optimisation is the smart alternative to grid search — it learns from every trial to choose the next hyperparameter configuration most likely to improve the model.