← AI Terminology
Boosting
Boosting is an ensemble method that sequentially trains weak learners (usually shallow decision trees), each one correcting the mistakes of its predecessor, combining them into a single strong predictor.
XGBoost, LightGBM, and CatBoost are the dominant boosting implementations used in production today.
XGBoost, LightGBM, and CatBoost are the dominant boosting implementations used in production today.
Why It Matters in AI
Boosting is the most consistently dominant algorithm on structured/tabular ML tasks — it wins more Kaggle competitions than any other method and is widely deployed in industry for credit scoring, fraud detection, and recommendation systems. XGBoost is often the first model data scientists reach for on new tabular problems, and ensemble solutions for tabular data almost always include a boosting variant.
Key Points
| Aspect | Description |
|---|---|
| XGBoost | Regularised gradient boosting with second-order gradients, pruning, parallelism — widely dominant |
| AdaBoost | Original boosting (1997): reweights misclassified samples so next tree focuses on them |
| CatBoost | Yandex: best native categorical feature handling, ordered boosting to prevent target leakage |
| LightGBM | Microsoft: histogram-based splits, leaf-wise growth — faster and more memory-efficient than XGBoost |
| Sequential | Each tree is trained on the residuals (errors) of all previous trees — not independent like bagging |
| Gradient boosting | Generalises AdaBoost: fits trees to pseudo-residuals (gradients of loss) — more flexible |
Simple Analogy
A debate team where each member studies the arguments the previous speaker got wrong. The first speaker covers the main points; the second focuses on what the first missed; each subsequent speaker zeroes in on remaining weaknesses. Together they cover far more ground than any one speaker could.
Common Usage Examples
xgb.XGBClassifier(n_estimators=500, learning_rate=0.05, max_depth=6).fit(X_train, y_train)lgb.LGBMRegressor(num_leaves=31, n_estimators=1000, learning_rate=0.05)for large tabular datasets- Kaggle: boosting ensembles win the majority of structured-data competitions
- SHAP feature importance:
shap.TreeExplainer(xgb_model).shap_values(X_test)— tree SHAP is exact for boosted trees - Credit scoring: LightGBM models scoring loan applicants in real time at major banks
Summary
In short: Boosting builds a strong model by sequentially correcting weak ones — XGBoost and LightGBM make it the reigning champion algorithm for structured data.