← AI Terminology
Bagging - Bootstrap Aggregating
Bagging (Bootstrap Aggregating) is an ensemble technique that trains multiple independent models on different random subsets of the training data (drawn with replacement) and aggregates their predictions.
Random Forest is the most famous bagging-based algorithm.
Random Forest is the most famous bagging-based algorithm.
Why It Matters in AI
Bagging reduces variance — the tendency of a model to overfit to the specific training set it saw. Because each model in the ensemble sees a different random sample, they make different errors; averaging their predictions cancels out individual mistakes. It's the reason Random Forests are among the most reliable out-of-the-box ML algorithms for tabular data.
Key Points
| Aspect | Description |
|---|---|
| Bootstrap | Each model trains on a random sample of n rows drawn with replacement from the training set |
| Aggregation | Classification: majority vote. Regression: mean of predictions |
| Parallelism | Each model is independent — easily parallelised across CPUs/GPUs |
| Key algorithm | Random Forest adds feature randomness on top of bagging — the dominant bagging variant |
| Out-of-bag (OOB) | Rows not sampled for a given tree serve as its built-in validation set — free evaluation |
| Contrast boosting | Bagging: parallel, reduces variance. Boosting: sequential, reduces bias. Both reduce error |
Simple Analogy
Ask 100 people to each estimate the weight of a mystery object after briefly inspecting it. Each person sees slightly different angles and has different biases. Average their answers and you get a far more accurate estimate than any one person — that's bagging. Crowd wisdom through variance reduction.
Common Usage Examples
BaggingClassifier(estimator=DecisionTreeClassifier(), n_estimators=100)in scikit-learnRandomForestClassifier(n_estimators=500, max_features='sqrt')— bagging + feature randomness- Out-of-bag score:
RandomForestClassifier(oob_score=True).fit(X, y).oob_score_ - Bagging for neural networks: train 5 identical architectures with different seeds, average softmax outputs
- Deep ensembles: bagging-like approach used by top-performing Kaggle image classification solutions
Summary
In short: Bagging trains multiple models on random data subsets and averages the results — reducing overfitting through the wisdom of crowds principle.