← AI Terminology
Stacking
Stacking (stacked generalisation) is an ensemble method that trains a meta-learner (blender) on the predictions of multiple base models — combining diverse base model outputs as features for a second-level model that learns how to best weight each base model's predictions.
It is one of the three main ensemble methods alongside bagging and boosting.
It is one of the three main ensemble methods alongside bagging and boosting.
Why It Matters in AI
Stacking is a systematic way to combine diverse model types: a decision tree, a linear model, and a neural network might each capture different patterns. Averaging their predictions is naive (equal weight); stacking lets a meta-learner discover the optimal weighting. On Kaggle, stacking or blending is responsible for the majority of winning solutions — combining 10–100 diverse models with a meta-learner routinely outperforms any individual model by a few percent.
Key Points
| Aspect | Description |
|---|---|
| Blending | Simpler variant: hold out a separate validation set for meta-learner training — faster but uses less data |
| Diversity | Different model types, feature subsets, or hyperparameters — high correlation between base models reduces gains |
| Kaggle use | Classic stacking: RF + GBM + NN + Logistic predictions → LightGBM meta-learner — top competition technique |
| Base models | Diverse set of level-0 models: RF, GBM, SVM, neural net, linear — diversity is key |
| Out-of-fold | Base models predict on held-out folds during CV — prevents meta-learner from memorising base model training predictions |
| Meta-learner | Level-1 model trained on out-of-fold predictions — logistic regression, LightGBM, simple NN |
Simple Analogy
A panel of analysts where each specialist (base model) provides a forecast, and a senior analyst (meta-learner) combines the forecasts based on their observed historical accuracy: "I weight the economist's GDP forecast at 40% and the quantitative model at 60% based on past accuracy." The senior analyst learns from experience, not just averaging.
Common Usage Examples
sklearn.ensemble.StackingClassifier(estimators=[("rf", rf), ("svm", svm)], final_estimator=LogisticRegression())sklearn.model_selection.cross_val_predict(base_model, X, y, cv=5, method='predict_proba')— generate OOF predictions- Kaggle stacking: train RF, XGB, NN → save OOF predictions → train LightGBM on OOF predictions
vecstack.stacking—stacking(models, X_train, y_train, X_test, regression=False, mode='oof_pred')- Multi-level: base models → level-1 meta → level-2 meta — diminishing returns past 2 levels
Summary
In short: Stacking trains a meta-learner on the outputs of diverse base models — learning the optimal combination rather than naive averaging, the ensemble technique responsible for most Kaggle competition winning solutions.