The Leaky ReLU function is a variant of the ReLU function, which introduces a small slope in the negative value interval, and solves the problem that the output of the ReLU function is zero in the negative value interval. The formula of the Leaky ReLU function is as follows:
scss
f(x) = max(ax, x)
Among them, a is a constant less than 1, called the leak coefficient. Generally, a takes a small value, such as 0.01.
The following is a Python sample code using the Leaky ReLU function:
python
import numpy as np
def leaky_relu(x, alpha=0.01):
return np.maximum(alpha*x, x)
# Example with a single value
x = -2
result = leaky_relu(x)
print(result) # output: -0.02
# Example using NumPy arrays
x_array = np.array([-2, -1, 0, 1, 2])
result_array = leaky_relu(x_array)
print(result_array) # Output: [-0.02 -0.01 0. 1. 2. ]
In the above example, we defined a leaky_relu function that takes an input value x and returns the computed result. You can set the value of the leak coefficient by specifying the alpha parameter, which defaults to 0.01. Then we used a single value and a NumPy array as an example, calculated the corresponding Leaky ReLU function value, and printed the output.
The Leaky ReLU function generates a non-zero output in the negative interval by introducing a small slope, avoiding the problem that the ReLU function outputs zero in the negative interval. This helps to solve the "dead" neuron problem of the ReLU function and provides a better flow of gradients.
The Leaky ReLU function is often used as the activation function of the deep learning model in practical applications, especially when the ReLU function does not perform well. By adjusting the value of the leak coefficient, the performance of the model can be affected. Usually, setting the leak coefficient to a small value, such as 0.01, can avoid its disadvantages while retaining the advantages of ReLU.