Random Forests Explained: Why Decision Trees Vote Better Together

Random Forests Explained: Why Decision Trees Vote Better Together

Decision trees are the friendliest model in machine learning: you can draw one on paper, explain it to anyone, and train one in three lines of code. Then you test it on new data and watch it fall apart. A single decision tree almost always overfits — it memorises the training data, quirks and noise included. The random forest is the fix the field converged on: grow hundreds of deliberately varied trees and let them vote. It is the most dependable first model for tabular data, it needs barely any tuning, and understanding why it works teaches you the single most important idea in applied ML — the bias-variance trade-off. This guide builds that understanding from the ground up, then walks through a complete scikit-learn workflow you can adapt to your own data.

The problem with a single clever tree

A decision tree learns by asking yes-or-no questions about your features: is income above 50k? Is the credit score below 700? Each answer splits the data, and the tree keeps splitting until every leaf is (ideally) pure — one class per leaf. The result reads like a flowchart, which is why students love it for coursework.

The trouble is that a fully grown tree is too good at its job. Given enough depth, it will isolate individual training points, drawing elaborate boundaries around noise, mislabelled rows and one-off outliers. Training accuracy hits 100 percent while test accuracy lags far behind — the textbook signature of overfitting. Change a few rows of the training data and you get a visibly different tree: the model has high variance. One tree is an overconfident expert who memorised last year's exam instead of learning the subject.

How a decision tree actually decides

Before the forest, the tree. At every node the algorithm searches all features and all possible thresholds for the split that best separates the classes, measured by impurity:

• Gini impurity = 1 − Σ(p²), where p runs over class proportions. A pure node (all one class) scores 0; a perfectly mixed binary node scores 0.5.
• Entropy is the alternative (−Σ p·log₂p); in practice the two behave almost identically.

A tiny example: a node holds 10 loan applications, 6 repaid and 4 defaulted. Gini = 1 − (0.6² + 0.4²) = 1 − (0.36 + 0.16) = 0.48. A candidate split sends 5 repaid one way (pure, Gini 0) and 1 repaid + 4 defaulted the other (Gini = 1 − (0.04 + 0.64) = 0.32). The weighted impurity after the split is (5/10)×0 + (5/10)×0.32 = 0.16 — a big drop from 0.48, so the tree takes that split. It repeats this greedily until a stopping rule fires: max depth reached, or too few samples left to split.

For regression the idea is identical with variance in place of impurity: splits minimise the variance inside each child node, and leaves predict the mean of their samples. Same tree, different scorekeeping.

Why deep trees memorise noise

Every split is chosen to fit the training data a little better, and nothing in a fully grown tree says "stop, that last split was fitting noise." With enough depth, the tree can give every training point its own leaf. The decision boundary becomes a jagged fence around individual points rather than a smooth separation of the underlying pattern.

This is the bias-variance trade-off in its purest form: a deep tree has low bias (it can represent almost anything) but high variance (it changes a lot with the data). Shallow trees have the opposite problem — stable but too crude. The random forest's insight is that you can keep the low bias of deep trees and average away the variance, provided the trees make different mistakes. Averaging only helps if the errors are uncorrelated — and that is what the forest's two sources of randomness are for.

The random forest fix: bagging plus random features

Each tree in the forest is trained differently in two deliberate ways:

1. Bootstrap sampling (bagging). Each tree gets a training set of the same size as the original, drawn with replacement. Statistically, each bootstrap sample contains about 63 percent of the unique data points — the rest are duplicates, and roughly 37 percent of the original data never appears in that tree's training set. So every tree sees a slightly different version of the world.
2. Random feature subsets. At each split, the tree may only consider a random subset of features (the square root of the feature count is the classic default for classification). This is the subtle, crucial part: without it, a dominant feature would be chosen at the root of every tree and all trees would look alike. Forcing different features at different splits decorrelates the trees, so their errors cancel instead of compounding.

Prediction is then democratic: majority vote for classification, average for regression. One overconfident friend memorised the exam; a poll of two hundred friends who each studied slightly different notes converges on the truth. That is the whole algorithm — bagging plus feature randomness plus voting — and it works so reliably that it remains the default first model for tabular data years after fancier methods arrived.

Build one end to end

Here is the complete workflow on scikit-learn's built-in wine dataset. It runs as-is in a notebook or a plain script:

from sklearn.datasets import load_wine
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score, classification_report

X, y = load_wine(return_X_y=True)
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

forest = RandomForestClassifier(n_estimators=200, random_state=42)
forest.fit(X_train, y_train)

pred = forest.predict(X_test)
print("Test accuracy:", round(accuracy_score(y_test, pred), 3))
print(classification_report(y_test, pred))

Set random_state so your results are reproducible — the forest is random by design, and you want the same randomness every run while you experiment. Now compare against a single fully grown tree to see the overfitting gap directly:

from sklearn.tree import DecisionTreeClassifier

tree = DecisionTreeClassifier(random_state=42)
tree.fit(X_train, y_train)

print("Tree  train accuracy:", round(tree.score(X_train, y_train), 3))
print("Tree  test  accuracy:", round(tree.score(X_test, y_test), 3))
print("Forest train accuracy:", round(forest.score(X_train, y_train), 3))
print("Forest test  accuracy:", round(forest.score(X_test, y_test), 3))

The single tree typically shows a large train-test gap; the forest's gap is far smaller. That gap — not the training score — is the number that matters.

The four knobs that actually matter

Random forests are famously insensitive to hyperparameters, but four are worth knowing:

• n_estimators — number of trees. More trees = more stable predictions, with diminishing returns past a few hundred. Cost grows linearly, and more trees never cause overfitting (they just converge). Start at 200–500 for final models.
• max_depth — the deepest any tree may grow. None (unlimited) is the default and usually fine inside a forest, since voting controls the variance. On noisy data, limiting depth to roughly 8–20 can help.
• min_samples_leaf — minimum samples in a leaf. Raising it from 1 to 3–5 smooths decision boundaries; it is the cheapest regularisation available.
• max_features — features considered per split. 'sqrt' is the classification default; smaller values make trees more diverse but individually weaker.

Tuning order: set n_estimators high enough that results stop changing, then tune max_depth and min_samples_leaf with cross-validation. That is genuinely most of it — if a tutorial tells you to grid-search twelve parameters for a random forest, it is overcomplicating.

Feature importance: useful, with a catch

During training the forest records how much each feature decreased impurity across all splits — the familiar feature importance scores:

import pandas as pd

importances = pd.Series(
    forest.feature_importances_, index=load_wine().feature_names
).sort_values(ascending=False)
print(importances.head(8).round(3))

This is genuinely useful for answering "which columns drive the prediction" — for sensor selection, for deciding what data to collect next, for sanity-checking the model against domain knowledge. But know the catch: impurity-based importance is biased toward high-cardinality features (a column with many unique values gets more chances to split) and splits credit between correlated features, so two equally useful correlated columns can each look half as important. For honest answers, use permutation importance: shuffle one column, measure how much the score drops, repeat. It is slower but far more trustworthy, and scikit-learn provides it in one function call.

Free validation: the out-of-bag score

Remember that each tree trains on only ~63 percent of the data. The remaining ~37 percent — the out-of-bag (OOB) samples — were never seen by that tree, so they act as a built-in test set. Aggregate each sample's prediction over just the trees that did not train on it, and you get a validation score without holding out any data:

forest_oob = RandomForestClassifier(
    n_estimators=200, oob_score=True, random_state=42
)
forest_oob.fit(X_train, y_train)
print("OOB score:", round(forest_oob.oob_score_, 3))

The OOB score tracks test accuracy surprisingly well. It is not a replacement for a proper held-out test set in a final report, but during experimentation it saves you from carving your small dataset into ever-smaller pieces.

Random forest versus the alternatives

Model	Strengths	Weaknesses	Reach for it when…
Single decision tree	Fully interpretable, fast, no scaling	Overfits badly	Teaching, or a baseline you can draw on a slide
Random forest	Robust default, handles mixed types, minimal tuning, no scaling needed	Memory-hungry, not interpretable as one model	Your first serious model on tabular data
Gradient boosting (XGBoost, LightGBM)	Often the best tabular accuracy	More hyperparameters, easier to overfit, slower to tune	The forest plateaus and you can afford tuning time
Logistic regression	Calibrated probabilities, interpretable coefficients	Assumes roughly linear relationships	You must explain why per feature, or need honest probabilities

Notice what the forest does not require: feature scaling. Splits are threshold comparisons, so monotonic transforms change nothing — one less preprocessing step than k-NN, SVMs or neural networks demand.

Mistakes students keep making

• Reporting training accuracy as the result. The forest will happily memorise the training set; only test/OOB numbers mean anything.
• Target leakage. Including a feature that is a consequence of the label — a "loan approved date" when predicting default, a post-treatment measurement when predicting disease. The model looks brilliant until deployment.
• Tuning on the test set. Every peek at the test score leaks information. Tune with cross-validation or OOB; touch the test set once, at the end.
• Ignoring class imbalance. On 95/5 data, predicting "always the majority class" scores 95 percent accuracy and is useless. Use class_weight="balanced", resample, and judge with precision, recall and F1 — not raw accuracy. (The evaluation-metrics guide linked below covers this properly.)
• Reading importance as causation. "The model relies on zip code" does not mean zip code causes the outcome — it may just correlate with the real driver.
• Forgetting the forest is still just pattern matching. It interpolates between training examples; it cannot reason about cases unlike anything it has seen.

FAQ

How many trees is enough?

One hundred is the traditional default and often adequate for exploration. For final models, 200–500 is the common range; watch the OOB error as you add trees and stop where it plateaus. Adding trees never hurts accuracy — only runtime.

Do I need to normalise or standardise features?

No. Tree splits compare feature values against thresholds, so scaling changes nothing about the model. This is a genuine practical advantage over distance- and gradient-based methods.

Can random forests handle missing values?

scikit-learn's classic implementation needs imputation first (median for numeric, most-frequent for categorical is fine for coursework). Some newer implementations handle missing values natively. Either way, think about why values are missing before filling them blindly.

Is it the same idea for regression?

Yes. The forest structure is identical; only the leaves change — they predict the mean of their samples instead of voting, and the final prediction is the average across trees.

Why is my forest slow or huge?

Training cost scales with trees × depth × data size, and 500 deep trees on a large dataset can use gigabytes of memory. Remedies: fewer trees, max_depth limits, n_jobs=-1 to parallelise across CPU cores, or train on a representative sample.

When should I move to gradient boosting?

When the random forest's performance plateaus and you have the time to tune carefully. Boosting often wins on tabular benchmarks, but it has more hyperparameters, overfits more readily, and takes longer to get right — earn it after the forest.

Limitations

• Not interpretable as a single model. You cannot show anyone "the" tree; explanation means importance scores, partial dependence plots or SHAP values — all approximations.
• Memory footprint. Hundreds of deep trees are heavy; forests do not belong on microcontrollers or in tight latency budgets without distillation or pruning.
• Poor extrapolation. A regression forest predicts a constant beyond the training data's range — dangerous for trends over time or out-of-distribution inputs.
• Correlated features dilute importance scores, and the forest does no clean automatic feature selection — do not skip thinking about your features.
• Wrong tool for perceptual data. On images, text and sequences, convolutional and transformer architectures outperform forests decisively. Forests shine on tabular data; use them there.

Suitable for students who want a dependable first classifier or regressor for coursework and tabular data projects, and who want to understand why ensembles work rather than treating them as magic.