← AI Terminology
Decision Tree
A decision tree is a supervised ML model that makes predictions by repeatedly splitting the data on feature thresholds, forming a tree of if/else rules that ends in a prediction at each leaf.
It is one of the oldest and most interpretable machine learning algorithms, and the building block for ensemble methods like Random Forest and Gradient Boosting.
It is one of the oldest and most interpretable machine learning algorithms, and the building block for ensemble methods like Random Forest and Gradient Boosting.
Why It Matters in AI
Decision trees are the most explainable non-trivial ML model — you can literally print the rules and hand them to a non-technical stakeholder. They handle mixed feature types (numeric and categorical) without scaling and require almost no preprocessing. Crucially, ensembles of hundreds of trees (XGBoost, LightGBM, Random Forest) consistently win tabular ML competitions and are widely deployed in finance, healthcare, and fraud detection.
Key Points
| Aspect | Description |
|---|---|
| Depth | Maximum number of splits from root to leaf; deeper = more complex, higher overfitting risk |
| Origin | ID3 (Quinlan, 1986) → C4.5 → CART — modern libraries implement CART |
| Pruning | Removing branches that add little predictive power to reduce overfitting |
| Ensembles | Random Forest (bagged trees), XGBoost / LightGBM / CatBoost (boosted trees) |
| Limitation | A single deep tree overfits easily; ensembles are almost always preferred for production |
| Split criterion | Gini impurity or entropy (classification); MSE (regression) — measures how well a split separates classes |
Simple Analogy
A decision tree is like a game of 20 Questions: at each step you ask the most informative yes/no question ("Is market cap > $10B?"), narrow down the possibilities, then ask the next best question. When you can't improve further, you make your best guess — that's the leaf.
Common Usage Examples
DecisionTreeClassifier(max_depth=5)in scikit-learn- Visualising a tree:
sklearn.tree.plot_tree(model, feature_names=cols) - XGBoost:
xgb.XGBClassifier(n_estimators=300, learning_rate=0.05)— ensemble of boosted trees - Credit scoring models in banks often use decision trees for regulatory explainability
- Feature importance from a tree:
model.feature_importances_ranks which splits mattered most
Summary
In short: A decision tree turns data into a flowchart of rules — simple enough to explain to anyone, and powerful enough (in ensembles) to beat neural networks on structured data.