ML Made Simple

The Decision-Making Process: A Tale of Decision Trees

Imagine you’re deciding whether to go outside. You start with a big question and break it down into smaller, more manageable ones, like “Is it raining?” or “Do I have homework?” Each question leads you down a path, eventually helping you reach a final decision. Now, let’s apply this approach to decision trees in machine learning.

Understanding Decision Trees

• Root Node: Begins with a primary question about your data.
• Internal Nodes: Break the primary question into smaller ones based on features.
• Branches: Each question leads to a decision branch based on answers.
• Leaf Nodes: Culminate in a final decision or prediction.

Types of Decision Trees

1. Decision Trees for Classification:
   • In classification, decision trees are used to partition the data into subsets based on features, ultimately assigning a class label to each subset.
   • At each node, the algorithm selects the feature that best separates the data based on certain criteria (e.g., Gini impurity or information gain).
2. Decision Trees for Regression:
   • In regression, decision trees predict a continuous target variable instead of a class label.
   • At each node, the algorithm chooses a feature and splits the data, aiming to minimize the variance of the target variable within the subsets.

Decision Trees Example with Gini Impurity

Let’s consider a dataset with features such as the number of rooms, area, and garage, and a binary target variable indicating whether the house price is high or low.

Number of Rooms	Area (sq. ft.)	Garage	Price (High/Low)
3	1500	Yes	Low
4	2000	No	Low
3	1800	Yes	Low
5	2500	Yes	High
4	1900	No	High
6	3000	Yes	High
5	2200	No	Low
4	2100	Yes	High
3	1600	No	Low
5	2300	No	High

Let’s say we want to split this dataset based on the feature “Number of Rooms.” We’ll calculate the Gini impurity at each step. The Gini impurity for a node is given by the formula:

where p_i is the probability of class i in the node. For a binary classification problem (High/Low price), k=2.

1. Initial Gini Impurity (before any split):
   • Total data points = 10
   • Number of High price = 5
   • Number of Low price = 5
   • Gini(all data)=1−((5/10)²+(5/10)²)=0.5
2. Split based on “Number of Rooms” (e.g., 4 rooms or less vs. more than 4 rooms):
   • Node 1 (Number of Rooms <= 4):
     • High price = 3
     • Low price = 4
     • Gini(Node 1)=1−((3/7)²+(4/7)²)
   • Node 2 (Number of Rooms > 4):
     • High price = 2
     • Low price = 1
     • Gini(Node 2)=1−((2/3)²+(1/3)²)
3. Calculate Weighted Gini Impurity after the split:

Weighted Gini=((Size of Node 1 / Total Size) × Gini(Node 1)) + ((Size of Node 2 / Total Size) × Gini(Node 2))

This is the Gini impurity for the split based on the “Number of Rooms” feature. The goal in building a decision tree is to find splits that minimize the Gini impurity, indicating a more homogenous set of data in each branch. The lower the Gini impurity, the better the split.

Now you may ask, why did algorithm choose the first split at “Number of Rooms” with value “4”?

• Consider All Unique Values of the Feature:
  • For the “Number of Rooms” feature, the algorithm identifies all unique values present in the dataset. Let’s say these values are 3,4,5, and 6.
• Evaluate Gini Impurity for Each Unique Value:
  • For each unique value, the algorithm considers it as a potential split point.
  • It calculates the Gini impurity for the resulting subsets after the split using that specific value.
  • The Gini impurity is calculated for each split point.
• Choose the Split Point with the Lowest Gini Impurity:
  • The algorithm selects the unique value that minimizes the Gini impurity for the resulting subsets.
  • This selected value becomes the optimal split point for the continuous feature.