ML Made Simple

Random Forest: A Council of Decision Trees

Imagine solving a problem with a diverse group of friends. Each friend suggests a different solution. You trust the consensus more than any single opinion. Random Forest in machine learning operates on a similar principle, combining the wisdom of multiple decision trees for better predictions.

Conceptualizing Random Forest

• Each Friend as a Tree: In a Random Forest, every decision tree is an independent problem solver.
• Diverse Perspectives: Trees focus on different aspects, adding variety to the solutions.
• Combining Votes: The collective decision from all trees forms the final prediction.
• The Randomness Factor: Each tree examines random subsets of data and features, ensuring a broad perspective.

Random Forest in Action

1. Random Forest for Classification:
   • A random forest is an ensemble of decision trees. It builds multiple decision trees and merges their predictions to improve overall accuracy and reduce overfitting.
   • Each tree is trained on a random subset of the data and a random subset of features at each split.
   • The final prediction is often determined by a majority vote among the trees.
2. Random Forest for Regression:
   • In regression, random forests combine the predictions of individual decision trees to provide a more robust and accurate prediction.
   • Instead of a majority vote, the final prediction is usually the average of the predictions from all the trees.

Random Forest Example

Number of Rooms	Area (sq. feet)	Has Garage	Price (Category)
3	1500	No	Affordable
4	2000	Yes	Expensive
2	1200	No	Affordable
5	2500	Yes	Expensive
4	1800	No	Affordable
3	1600	Yes	Expensive

Number of Rooms	Area (sq. feet)	Has Garage	Price (Category)
4	2000	Yes	Expensive
3	1600	Yes	Expensive

Number of Rooms	Area (sq. feet)	Has Garage	Price (Category)
2	1200	No	Affordable
5	2500	Yes	Expensive

Number of Rooms	Area (sq. feet)	Has Garage	Price (Category)
4	1800	No	Affordable
3	1500	No	Affordable

Now, three decision trees are built, each on one of these random subsets, considering a random subset of features at each split.

For example, the first decision tree might be built using Random Subset 1 and considering only the “Number of Rooms” and “Has Garage” features at each split. The second decision tree might be built using Random Subset 2 and considering only the “Area” and “Number of Rooms” features. The third decision tree might be built using Random Subset 3 and considering only the “Area” and “Has Garage” features.

When making a prediction for a new data point (e.g., a house with specific features), the predictions from all three trees are combined. For classification problems, the mode of the predictions is often taken, and for regression problems, the average of the predictions is commonly used.

This process helps to create an ensemble of diverse trees(also known as bagging), reducing overfitting and improving the model’s generalization performance.

Benefits of Random Forest

• Reduced Overfitting: The ensemble approach mitigates overfitting, a common issue with individual trees.
• Improved Generalization: By considering various aspects, Random Forest adapts well to new data.
• Versatility: Effective for both classification and regression tasks.