ML Made Simple

What is a loss function?

Loss functions, also known as objective functions or cost functions, play a crucial role in training neural networks. These functions measure the difference between the predicted output of a model and the true target values, providing a signal for the model to adjust its parameters during the training process.

In this article, I will just demonstrate how the loss is computed. All of the following in other articles: Once the loss is calculated, the gradient of the loss with respect to the network parameters is computed through back-propagation. The gradients are then used to update the parameters using an optimization algorithm. The process is repeated for multiple batches of training data until the model converges to a set of parameters that minimize the overall loss on the training data.

MSE(mean square error) / L2 loss:

Mean Squared Error (MSE) is a way to measure how different a set of numbers are from each other. It’s often used in the context of evaluating how well a prediction or estimate matches the actual values.

1. Squared Difference:
   • For each pair of corresponding values (one predicted and one actual), find the difference between them.
   • Square each of these differences.
2. Average:
   • Find the average (mean) of all the squared differences.

So, MSE is basically the average of the squared differences between predicted and actual values. It gives you a single number that represents how far off your predictions are from the actual values. The smaller the MSE, the closer the predictions are to the actual values.

Now let’s see how the MSE / L1 loss is computed:

def mean_squared_error(actual, predicted):
    # Calculate squared differences
    squared_diff = (actual - predicted) ** 2
    
    # Calculate mean squared error
    mse = sum(squared_diff) / len(actual)
    
    return mse

# Example usage:
actual_values = [3.0, 5.0, 2.0, 7.0]
predicted_values = [2.5, 5.5, 1.8, 6.5]

# Calculate MSE Loss
loss = mean_squared_error(actual_values, predicted_values)

# Print the result
print(f"Mean Squared Error (MSE) Loss: {loss}")

MAE(mean absolute error) / L1 loss:

Mean Absolute Error (MAE) is another way to measure how different a set of numbers are from each other.

1. Absolute Difference:
   • For each pair of corresponding values (one predicted and one actual), find the absolute difference between them. The absolute difference is always positive.
2. Average:
   • Find the average (mean) of all these absolute differences.

So, MAE is the average of the absolute differences between predicted and actual values. It gives you a single number that represents the average magnitude of errors. The smaller the MAE, the closer the predictions are, on average, to the actual values.

Now let’s see how MAE / L1 loss is computed:

def mean_absolute_error(actual, predicted):
    # Calculate absolute differences
    absolute_diff = abs(actual - predicted)
    
    # Calculate mean absolute error
    mae = sum(absolute_diff) / len(actual)
    
    return mae

# Example usage:
actual_values = [3.0, 5.0, 2.0, 7.0]
predicted_values = [2.5, 5.5, 1.8, 6.5]

# Calculate MAE Loss
loss = mean_absolute_error(actual_values, predicted_values)

# Print the result
print(f"Mean Absolute Error (MAE) Loss: {loss}")

Cross Entropy (CE) loss:

Cross-Entropy Loss, often used in the context of machine learning, is a way to measure how well a set of predictions matches the actual outcomes, especially in classification problems.

1. Actual Outcome Representation:
   • Imagine you have different categories (like cats and dogs).
   • Each actual outcome (what really happened) is represented as a probability distribution. For example, if it’s a cat, the distribution might be [1, 0] (100% chance of being a cat, 0% chance of being a dog).
2. Predicted Outcome Representation:
   • Similarly, your model makes predictions, and each prediction is represented as a probability distribution.
3. Cross-Entropy Calculation:
   • For each category, multiply the actual probability by the logarithm of the predicted probability for that category.
   • Sum up these values for all categories.
4. Negative Log-Likelihood:
   • Take the negative of the result from step 3.
5. Average:
   • Find the average (mean) over all examples.

So, Cross-Entropy Loss measures how well the predicted probabilities match the actual probabilities. If the predictions are close to the actual outcomes, the Cross-Entropy Loss is low; if they differ a lot, the loss is higher. It’s a way to quantify how well a model is performing in terms of its predictions in a classification task.

Now let’s see how cross entropy (CE) loss is computed:

import math

def multi_class_cross_entropy_loss(y, y_hat):
    # Avoid log(0) by adding a small epsilon value
    epsilon = 1e-15
    
    # Calculate the loss for each class and sum them up
    total_loss = -sum(y_i * math.log(y_hat_i + epsilon) for y_i, y_hat_i in zip(y, y_hat))
    
    return total_loss

# Example usage:
actual_probabilities = [0, 1, 0]
predicted_probabilities = [0.2, 0.7, 0.1]

# Calculate Multi-Class Cross-Entropy Loss
loss = multi_class_cross_entropy_loss(actual_probabilities, predicted_probabilities)

# Print the result
print(f"Multi-Class Cross-Entropy Loss: {loss}")

Focal Loss:

Focal Loss is a modification to the standard Cross-Entropy Loss, designed to address the problem of class imbalance in certain machine learning tasks, especially in object detection.

1. Class Imbalance Issue:
   • In some tasks, like object detection, there might be a lot of background (common) examples and fewer examples of rare objects. This can lead to a model that becomes biased towards predicting the common class.
2. Focusing on Hard Examples:
   • Focal Loss gives more importance to hard-to-classify examples, or the examples where the model’s prediction is far from the actual value. It does this by introducing a modulating factor that reduces the contribution of easy examples.
3. Modulating Factor:
   • The modulating factor, often denoted as (1−y^)γ, is multiplied with the standard Cross-Entropy Loss term. It makes the loss contribution smaller for well-classified examples (y^ close to 1) and larger for misclassified examples (y^ far from 1).
4. Tuning with Hyperparameter:
   • The hyperparameter γ allows you to adjust the strength of the focusing. A higher γ gives more emphasis on hard examples.

So, Focal Loss is a tweak to Cross-Entropy Loss that helps the model pay more attention to challenging examples, addressing issues of class imbalance and improving performance on rare classes.

Now let’s see how focal loss is computed:

import math

def focal_loss(y, y_hat, gamma=2):
    # Avoid log(0) by adding a small epsilon value
    epsilon = 1e-15
    
    # Calculate the Focal Loss for each class and sum them up
    total_loss = -sum((1 - y_hat_i) ** gamma * y_i * math.log(y_hat_i + epsilon) for y_i, y_hat_i in zip(y, y_hat))
    
    return total_loss

# Example usage:
actual_probabilities = [0, 1, 0]
predicted_probabilities = [0.2, 0.7, 0.1]

# Calculate Focal Loss
loss = focal_loss(actual_probabilities, predicted_probabilities)

# Print the result
print(f"Focal Loss: {loss}")

Triplet Loss:

Triplet Loss is a concept used in tasks like face recognition or similarity learning.

1. Triplets:
   • In each training step, the model is presented with three examples:
     • Anchor (A): The reference example.
     • Positive (P): An example that should be similar to the anchor.
     • Negative (N): An example that should be dissimilar to the anchor.
2. Loss Calculation:
   • The loss is calculated based on the distances between these examples in the feature space.
   • The model is encouraged to make the distance between the anchor and positive examples small (to make them similar) and the distance between the anchor and negative examples large (to make them dissimilar).
3. Triplet Loss Function:
   • As shown in the formula above, where d(A,P) represents the distance between embedding A and P, and α is a margin that defines how much closer the positive example should be compared to the negative example.

Triplet Loss helps a model understand the relative distances between examples, ensuring that similar examples are close together and dissimilar examples are far apart in the learned feature space. This is useful in applications where recognizing similarities is crucial, such as face recognition or recommendation systems.

Now let’s see how triplet loss is computed:

import math

def euclidean_distance(x, y):
    # Calculate Euclidean distance between vectors x and y
    return math.sqrt(sum((xi - yi) ** 2 for xi, yi in zip(x, y)))

def triplet_loss(anchor, positive, negative, alpha=0.2):
    # Calculate distances
    distance_positive = euclidean_distance(anchor, positive)
    distance_negative = euclidean_distance(anchor, negative)
    
    # Calculate triplet loss
    loss = max(distance_positive - distance_negative + alpha, 0)
    
    return loss

# Example usage:
anchor_example = [1.0, 2.0, 3.0]
positive_example = [1.2, 1.8, 3.2]
negative_example = [4.0, 5.0, 6.0]

# Calculate Triplet Loss
loss = triplet_loss(anchor_example, positive_example, negative_example)

# Print the result
print(f"Triplet Loss: {loss}")

Appropriate Use Cases for Each Loss Function:

1. Mean Squared Error (MSE):
   • Example: Housing Price Prediction
   • Use Case: When predicting house prices, MSE is often used as a loss function. The model aims to minimize the squared differences between predicted and actual house prices.
2. Mean Absolute Error (MAE):
   • Example: Weather Forecasting
   • Use Case: In weather forecasting, MAE can be used to measure the accuracy of predicted temperatures. The absolute differences between predicted and actual temperatures are averaged.
3. Cross Entropy Loss:
   • Example: Image Classification
   • Use Case: In image classification tasks, cross entropy loss is commonly used. The model is trained to predict the correct class label, and the cross entropy loss penalizes deviations from the true distribution of classes.
4. Focal Loss:
   • Example: Object Detection (especially with imbalanced classes)
   • Use Case: In object detection tasks, focal loss is useful when dealing with imbalanced datasets. It helps the model focus more on hard-to-classify examples, improving performance on rare classes.
5. Triplet Loss:
   • Example: Face Recognition
   • Use Case: In face recognition, triplet loss is often employed. The model is trained to ensure that the distance between embeddings of the same person (positive pair) is minimized, while the distance between embeddings of different people (negative pair) is maximized.