ML Made Simple

What is an Activation Function?

Activation functions determine which neurons in a neural network will be activated or deactivated. They play a pivotal role in influencing the output of the network.

Purpose of Activation Functions

These functions introduce non-linearity in the relationship between weights and input data. Non-linear activation functions are crucial in managing issues like vanishing or exploding gradients during backpropagation, which are essential for effectively training deep networks.

Types of Activation Functions

• ReLU (Rectified Linear Unit):
  • Function: Keeps positive numbers unchanged, turns negative numbers to zero.
  • Analogy: Like an on-off switch, letting only positive aspects through.
• Sigmoid:
  • Function: Converts any number to a value between 0 and 1.
  • Analogy: A compressor, fitting everything neatly between 0 and 1.
• Tanh (Hyperbolic Tangent):
  • Function: Converts any number to a value between -1 and 1, centering around 0.
  • Analogy: Similar to Sigmoid, but centers values around 0.

Choosing the Right Activation Function: ReLU, Sigmoid, or TanH?

While ReLU is my personal favorite, the choice of activation function depends on the dataset and application. Let’s explore this through a Python code example where we compare the performance of a Multi-Layer Perceptron (MLP) classifier with these three activation functions on a synthetic dataset for binary classification.

Python Code: Comparing Activation Functions

Using numpy, matplotlib, and sklearn, we train an MLP classifier on a synthetic dataset and evaluate its performance with ReLU, Sigmoid, and TanH activation functions.

import numpy as np
import matplotlib.pyplot as plt
from sklearn.datasets import make_classification
from sklearn.model_selection import train_test_split
from sklearn.neural_network import MLPClassifier
from sklearn.metrics import accuracy_score

# Generate a synthetic dataset
X, y = make_classification(n_samples=1000, n_features=20, n_informative=10, n_redundant=5, random_state=42)
# Split the dataset into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Function to train and evaluate a neural network
def train_and_evaluate(activation_function):
    # Create a multi-layer perceptron classifier with the specified activation function
    clf = MLPClassifier(hidden_layer_sizes=100, activation=activation_function, max_iter=500, learning_rate_init=0.001, random_state=42)
    
    # Train the classifier
    clf.fit(X_train, y_train)
    
    # Predict on the test set
    y_pred = clf.predict(X_test)
    
    # Return accuracy
    return accuracy_score(y_test, y_pred)


accuracy_relu = train_and_evaluate('relu')
accuracy_sigmoid = train_and_evaluate('logistic')
accuracy_tanh = train_and_evaluate('tanh')

Results: ReLU vs. Sigmoid vs. TanH

In our experiment, ReLU shows superior performance over Sigmoid and TanH for binary classification on randomly generated data. However, it’s advisable to experiment with different activation functions for varied applications.