← AI Terminology
Gradient Boosting (XGBoost, LightGBM)
Gradient boosting is an ensemble method that builds a strong predictor by sequentially adding weak learners (usually shallow decision trees), each one trained to minimise the residual loss from all previous learners combined.
XGBoost and LightGBM are the dominant gradient boosting implementations — consistently winning structured/tabular ML competitions.
XGBoost and LightGBM are the dominant gradient boosting implementations — consistently winning structured/tabular ML competitions.
Why It Matters in AI
Gradient boosting is the most consistently dominant algorithm for tabular (structured) data. On Kaggle, the majority of winning solutions for tabular problems use XGBoost or LightGBM. In industry, it powers credit scoring, fraud detection, recommendation systems, and click-through-rate prediction at Google, Amazon, and banks worldwide. Neural networks rarely beat gradient boosting on clean tabular data without significant engineering.
Key Points
| Aspect | Description |
|---|---|
| XGBoost | Regularised GBDT: second-order gradients, L1/L2 regularisation, column subsampling |
| CatBoost | Ordered boosting + native categorical handling — best at avoiding target leakage |
| LightGBM | Histogram-based leaf-wise growth: 5–20× faster than XGBoost on large datasets |
| Core idea | Each new tree is trained on the negative gradient of the loss — fits residual errors directly |
| Shrinkage | Multiply each tree by a learning rate (0.01–0.1) — slower learning, better generalisation |
| Hyperparameters | n_estimators, max_depth (3–6 typical), learning_rate, subsample, colsample_bytree |
Simple Analogy
Each new tree in gradient boosting is a specialist called in to fix the mistakes of the team so far. The first tree handles the easy cases; the second focuses on what the first got wrong; each specialist addresses the remaining errors of all predecessors. Together, they cover the full problem space far better than any generalist.
Common Usage Examples
xgb.XGBClassifier(n_estimators=500, max_depth=6, learning_rate=0.05, subsample=0.8)lgb.LGBMRegressor(num_leaves=31, n_estimators=1000, min_child_samples=20)- Early stopping:
xgb.train(params, dtrain, evals=[(dval,'val')], early_stopping_rounds=50) - SHAP with XGBoost:
explainer = shap.TreeExplainer(model); shap.summary_plot(explainer(X)) - Optuna HPO: tune
max_depth,learning_rate,num_leaveswith 100 Bayesian trials
Summary
In short: Gradient boosting builds an ensemble by sequentially fixing errors — XGBoost and LightGBM are the reigning algorithms for tabular ML, winning more competitions than any other approach.