ML Made Simple

The Role of Back-Propagation in Neural Networks

This post builds on our previous discussions about loss computation, gradient calculation, and weight updates. If you’re new to these concepts, reviewing the earlier articles will enhance your understanding.

Understanding Back-Propagation

Back-propagation is the process of fine-tuning the weights of a neural network to minimize loss. It involves:

1. Feedback on Errors: The network learns from its mistakes, similar to a teacher guiding a student. Each layer is informed how much it contributed to the overall error.
2. Adjusting Weights: Weights are modified layer by layer in reverse order to reduce the error. It’s akin to fine-tuning a radio for better reception.
3. Iterative Improvement: The network iteratively adjusts its weights to improve its predictive accuracy, akin to a student learning from mistakes and improving over time.

Code Implementation

Experiment with this code to see how back-propagation works. Try using different optimizers (SGD, Momentum, RMSProp, Adam) to observe how they affect convergence.

import numpy as np

input_feature_1 = np.random.rand(100,1)
input_feature_2 = np.random.rand(100,1)
input_ground_truth = np.random.rand(100,1)

weight_1 = 2#intital random weight for input_feature_1
weight_2 = 3#initial random weight for input_feature_2
print("INITIAL WEIGHTS: ",weight_1," ",weight_2)

#to converge the loss to min, run for 10 epochs
epoch = 10
for _ in range(epoch):
#y = w1xI1 + w2xI2; w1 = 2 and w2 = 3
    output_feature_predicted = weight_1*input_feature_1 + weight_2*input_feature_2

    #L1/MSE loss or regression loss
    loss = np.mean((output_feature_predicted - input_ground_truth)**2)
    print("LOSS:", loss)#something around 6.12, which is huge, should be less than 1% or 0.01

    #now we try to minimize this loss by calculating gradient of weights(w1, w2 here)
    gradient_w1 = 2 * np.mean((output_feature_predicted - input_ground_truth) * input_feature_1)
    gradient_w2 = 2 * np.mean((output_feature_predicted - input_ground_truth) * input_feature_2)

    learning_rate = 0.1
    #SGD
    weight_1 = weight_1 - learning_rate*gradient_w1
    weight_2 = weight_2 - learning_rate*gradient_w2
    print("UPDATED WEIGHTS: ",weight_1," ",weight_2)

The Impact of Back-Propagation

• Error Minimization: The process aims to reduce the loss (error) iteratively, improving the model’s predictions.
• Learning Efficiency: Back-propagation enhances the network’s ability to learn from data, akin to a student improving problem-solving skills with guidance.