← AI Terminology
Ensemble Learning
Ensemble learning is the practice of combining predictions from multiple models to produce a single, more accurate prediction than any individual model could achieve alone.
It exploits the fact that diverse models make different errors, which cancel out when averaged.
It exploits the fact that diverse models make different errors, which cancel out when averaged.
Why It Matters in AI
Ensemble methods have won more machine learning competitions than any other technique. They reduce both variance (bagging) and bias (boosting) and are robust to any single model's failure modes. In practice, the top Kaggle solutions almost always ensemble multiple model types. Even in deep learning, ensembling multiple independently trained models typically yields 1–3% additional accuracy beyond the best single model.
Key Points
| Aspect | Description |
|---|---|
| Voting | Hard voting (majority class) or soft voting (average probabilities) — simplest ensemble |
| Bagging | Train models on bootstrap samples; average predictions — Random Forest is the canonical example |
| Boosting | Train models sequentially to correct each other's errors — XGBoost, LightGBM |
| Stacking | Train a meta-model on the predictions of base models — often the strongest ensemble technique |
| Diversity | More diverse models benefit more from ensembling — different architectures, features, seeds |
| Limitation | 2–10× inference cost and complexity; often a diminishing-returns exercise |
Simple Analogy
A panel of judges instead of one judge: different judges have different biases and blind spots. When they vote, systematic individual errors cancel out and the panel's collective decision is more reliable than any one judge. The more independent their perspectives, the better the ensemble.
Common Usage Examples
- Soft voting:
VotingClassifier(estimators=[...], voting='soft')in scikit-learn - Stacking:
StackingClassifier(estimators=[rf, xgb, lgbm], final_estimator=LogisticRegression()) - Test-time augmentation (TTA): ensemble predictions over multiple augmented versions of each test image
- Kaggle Gold: most winning solutions are ensembles of XGBoost + LightGBM + neural networks + stacking
- Deep ensemble: train 5 ResNets with different seeds — average softmax outputs for calibrated uncertainty
Summary
In short: Ensemble learning combines multiple models so their errors cancel out — the technique that wins competitions and improves reliability at the cost of inference speed.