← AI Terminology
Random Forest
A random forest is an ensemble learning algorithm that trains many decision trees on random subsets of the data and features, then aggregates their predictions (majority vote for classification, average for regression) — improving accuracy and reducing overfitting compared to any single decision tree.
It remains one of the most effective and widely used algorithms for structured/tabular data.
It remains one of the most effective and widely used algorithms for structured/tabular data.
Why It Matters in AI
Decision trees overfit: they memorise training data and perform poorly on test data. Random forests solve this via two sources of randomness: (1) bootstrap sampling — each tree trains on a random sample of rows; (2) feature randomness — each split considers only a random subset of features. The diversity between trees ensures their errors are uncorrelated — when averaged, errors cancel and signal accumulates. For most tabular ML problems without hyperparameter tuning, random forests are a strong, reliable baseline.
Key Points
| Aspect | Description |
|---|---|
| Bootstrap | Each tree trains on a bootstrap sample (random sample with replacement) of the training set |
| OOB error | Out-of-bag error: each tree evaluated on samples not in its bootstrap — free validation estimate |
| vs XGBoost | Random forest: parallel, simpler tuning; XGBoost: sequential boosting, often more accurate on tabular |
| Feature subset | At each split, consider only √n_features (classification) or n_features/3 (regression) features |
| Hyperparameters | n_estimators (more = better, up to a point), max_depth, max_features, min_samples_leaf |
| Feature importance | Measure how much each feature reduces impurity across all trees — useful for feature selection |
Simple Analogy
A jury system: instead of relying on one expert's judgement (a single tree), you select a diverse jury of 100 people (100 trees), each with slightly different backgrounds and information. The majority verdict is far more reliable than any individual — errors are random and cancel out, while correct reasoning aligns.
Common Usage Examples
sklearn.ensemble.RandomForestClassifier(n_estimators=200, max_features='sqrt', n_jobs=-1)model.fit(X_train, y_train)— parallelised training across n_jobs CPU cores- Feature importance:
model.feature_importances_— returns array of relative importance per feature - OOB score:
RandomForestClassifier(oob_score=True); model.oob_score_— free validation accuracy - Imbalanced data:
RandomForestClassifier(class_weight='balanced')— weight minority class
Summary
In short: Random forests aggregate many diverse decision trees trained on random data and feature subsets — a robust, high-accuracy ensemble that is a reliable baseline for virtually any tabular classification or regression problem.