ML Made Simple

Addressing Overfitting with Regularization

Following our discussions on underfitting and overfitting, let’s explore L1 and L2 regularization, effective techniques for enhancing model generalization.

Lasso (L1) Regularization: The Magic Organizer

• What is Lasso?: Imagine decluttering a room; Lasso does the same for models. It simplifies by setting some feature coefficients to zero, focusing only on crucial elements.
• Operation:
  • Promotes sparsity in models.
  • Acts as a natural feature selector.
• Use Cases:
  • Ideal for high-dimensional datasets.
  • Effective when many features are redundant or irrelevant.
• Example: In computer vision, Lasso helps focus on relevant pixels, ignoring the less significant ones.

Ridge (L2) Regularization: The Balanced Bridge

• What is Ridge?: Think of constructing a balanced bridge. Ridge ensures no single feature (pillar) becomes overly dominant, maintaining a harmonious model structure.
• Operation:
  • Encourages small but non-zero weights for all features.
  • Addresses multicollinearity.
• Use Cases:
  • Suitable when all features contribute but need controlled influence.
  • Beneficial in multicollinear scenarios in linear regression.
• Example: In financial modeling, Ridge handles correlated financial indicators effectively.

The Mechanism Behind L1 Inducing Sparsity

The key mechanism behind the sparsity effect is the nature of the optimization process when using L1 regularization. This is because the absolute value function is not differentiable at zero. The derivative of L₂ is 2 * weight. The derivative of L₁ is k (a constant, whose value is independent of weight).

You can think of the derivative of L₁ as a force that subtracts some constant from the weight every time. L₁ has a discontinuity at 0, which causes subtraction results that cross 0 to become zeroed out. For example, if subtraction would have forced a weight from +0.1 to -0.2, L₁ will set the weight to exactly 0. There we go, L₁ zeroed out the weight.

Regularization Strength (λ) and its Optimization

The regularization strength (λ) is a hyperparameter that needs to be tuned during the model training process. The choice of λ depends on the specific dataset and the desired trade-off between fitting the training data and preventing overfitting. Cross-validation is often used to find the optimal value of λ that generalizes well to unseen data.

How do we find optimal value of Regularization Strength (λ)?

Using k-fold cross-validation to identify the best possible value of the regularization parameter (λ) involves training and evaluating the model for different values of λ across multiple folds. The value of λ that results in the best average performance across all folds is typically considered the optimal choice. Here are the general steps:

Define a Range of λ: Determine a range of values for λ that you want to explore. This range should cover a spectrum from very small values (little regularization) to very large values (strong regularization).

Split Data into K Folds: Use k-fold cross-validation to split your dataset into k folds.

Iterate Over λ Values:

• For each λ in your defined range:
  • For each fold in the k-fold cross-validation:
    • Train the model using the training data from the current fold.
    • Validate the model on the validation data from the current fold.
    • Calculate a performance metric (e.g., accuracy, mean squared error).
  • Calculate the average performance metric across all folds for the current λ.

Choose Optimal λ: Identify the λ that resulted in the best average performance across all folds.Train Final Model: Train the final model using the entire dataset and the chosen optimal λ.

Upcoming Topics

Future articles will dive deeper into techniques like feature engineering and ensemble methods to further address model overfitting and underfitting.