← AI Terminology
Gaussian Process
A Gaussian Process (GP) is a probabilistic model that defines a distribution over functions — instead of predicting a single value, it predicts a mean and uncertainty (variance) at every input point, consistent with a multivariate Gaussian distribution.
It is the theoretically grounded uncertainty-quantifying alternative to neural networks for small-to-medium datasets.
It is the theoretically grounded uncertainty-quantifying alternative to neural networks for small-to-medium datasets.
Why It Matters in AI
GPs provide calibrated uncertainty: they don't just say "the prediction is 3.5" — they say "3.5 ± 0.8 (95% CI)." This makes them invaluable for Bayesian optimisation (where you need uncertainty to decide where to sample next), scientific applications (where uncertainty bounds matter), and active learning. Their weakness is cubic scaling with dataset size — largely restricting them to smaller problems.
Key Points
| Aspect | Description |
|---|---|
| Prior | Before seeing data: mean function + kernel specify what functions are plausible |
| Scaling | O(n³) training, O(n²) memory — impractical beyond ~10,000 points without approximations |
| Posterior | After seeing data: updated distribution over functions — exact Bayesian inference |
| Sparse GP | Inducing points approximate full GP — enables larger datasets |
| Applications | Bayesian optimisation, time series, geostatistics, active learning, surrogate modelling |
| Kernel function | Defines similarity structure between inputs — RBF, Matérn, periodic — encodes prior assumptions |
Simple Analogy
Drawing a curve through a few measured points, but instead of a single best-fit line, drawing a probability band — wide where data is scarce, narrow where data is dense. A GP is the mathematically principled version of this: a full probability distribution over all possible smooth curves consistent with the data.
Common Usage Examples
sklearn.gaussian_process.GaussianProcessRegressor(kernel=RBF())— small-dataset regression with uncertaintygpytorch.models.ExactGP— PyTorch GP with GPU acceleration and sparse approximations- Bayesian optimisation: GP models objective function; acquisition function uses GP uncertainty to pick next trial
botorch.fit_gpytorch_mll(mll)— fit a GP for Bayesian optimisation with BoTorch- Kriging in geostatistics: GP interpolation of spatial data (soil composition, ore grade estimation)
Summary
In short: A Gaussian Process is a probabilistic model that predicts a full uncertainty distribution over function values — the principled tool when you need calibrated uncertainty alongside predictions.