ML Made Simple

Tackling Overfitting and Underfitting with K-Fold Cross Validation

Following our exploration of bias, variance, underfitting, and overfitting, let’s dive into techniques like K-Fold Cross Validation to address these challenges.

What is K-Fold Cross Validation?

Imagine you have a new video game and you want to see how good you are at it. Instead of playing it once and deciding your skill level based on that single game, you decide to play it multiple times in different rounds to get a more accurate measure of your overall performance.

Now, k-fold cross-validation is like breaking down these rounds in a systematic way:

1. Divide the Rounds (Folds):
   • You decide to play the game, let’s say, 5 times. This means you have 5 rounds or folds.
2. Train-Test Split in Each Round:
   • In each round, you divide your gameplay into two parts: training and testing.
   • You practice and get better during the training part, and then you test your skills in the other part to see how well you’ve learned.
3. Average the Results:
   • After all 5 rounds, you take the scores from each testing part and calculate the average.
   • This average gives you a more reliable measure of your overall performance, reducing the chance that a single lucky or unlucky game influenced your judgment.
4. So, in the context of machine learning:
   • Training: The model learns from a portion of the dataset (training set) in each round.
   • Testing: The model is tested on a different portion (test set) in each round.
   • Average Performance: The average of the performance in all rounds gives a more robust estimate of how well the model might perform on new, unseen data.

Types of K-Fold Cross Validation

• K-Fold Cross-Validation:
  • The most common form is k-fold cross-validation.
  • The dataset is divided into k equally sized folds.(k=5/10/any value of choice)
  • The model is trained k times, each time using k-1 folds for training and the remaining fold for validation.
• Leave-One-Out Cross-Validation (LOOCV):
  • A special case of k-fold cross-validation where k is set to the number of samples in the dataset. (k=N)
  • The model is trained and validated for each sample individually, using all other samples for training.

Implementing K-Fold CV and LOOCV

1. Steps in K-Fold Cross-Validation:
   • Split Data into Folds:
     • Divide the dataset into k equally sized folds (subsets). Each fold is approximately the same size, and the data is divided without shuffling.
   • Iterate Through Folds:
     • For each iteration (k times):
       • Use k-1 folds for training the model.
       • Use the remaining fold for validation (testing) the model.
       • Train and evaluate the model k times, each time using a different fold for validation.
   • Performance Metric Calculation:
     • Calculate the performance metric (e.g., accuracy, mean squared error) for each iteration, resulting in k performance values.
   • Average Performance:
     • Calculate the average performance across all iterations to obtain a more robust estimate of the model’s performance.
   • Use the Model:
     • Once the model has been trained and validated, it can be used on new, unseen data.
2. Steps in Leave-One-Out Cross-Validation (LOOCV):
   • Iterate Through Samples:
     • For each sample in the dataset (N times, where N is the number of samples):
       • Use N-1 samples for training the model.
       • Use the remaining sample for validation (testing) the model.
       • Train and evaluate the model N times, each time using a different sample for validation.
   • Performance Metric Calculation:
     • Calculate the performance metric (e.g., accuracy, mean squared error) for each iteration, resulting in N performance values.
   • Average Performance:
     • Calculate the average performance across all iterations to obtain a more robust estimate of the model’s performance.
   • Use the Model:
     • Once the model has been trained and validated, it can be used on new, unseen data.

Code Example: Demonstrating K-Fold CV

This Python example showcases the significance of k-fold CV using a random forest classifier on a hypothetical dataset.

import numpy as np
from sklearn.model_selection import train_test_split, cross_val_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.metrics import accuracy_score

# Generate a hypothetical dataset
np.random.seed(42)
X = np.random.rand(100, 5)  # 100 samples with 5 features
y = np.random.randint(2, size=100)  # Binary classification labels

# Split the dataset into a single train-test split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Train a random forest classifier on the single split dataset
model_single_split = RandomForestClassifier(random_state=42)
model_single_split.fit(X_train, y_train)
y_pred_single_split = model_single_split.predict(X_test)
accuracy_single_split = accuracy_score(y_test, y_pred_single_split)

# Perform k-fold cross-validation
k = 5  # Number of folds
model_cross_val = RandomForestClassifier(random_state=42)
cross_val_scores = cross_val_score(model_cross_val, X, y, cv=k, scoring='accuracy')

# Calculate the average cross-validation accuracy
average_cv_accuracy = np.mean(cross_val_scores)

# Display the results
print(f"Accuracy on a single split dataset: {accuracy_single_split:.2f}")
print(f"Average accuracy with {k}-fold cross-validation: {average_cv_accuracy:.2f}")

Visualizing K-Fold CV Performance

This graph displays the result on 100 samples of 5 features. Now you can imagine the importance of k-fold CV where features are in abundance and number of samples are greater.

Next Steps

Future articles will focus on specific techniques like L1/L2 regularization and feature engineering to further mitigate overfitting and underfitting.