The ReLU function (Rectified Linear Unit) is a commonly used activation function that maps negative values to zero and keeps positive values unchanged. The formula of the ReLU function is as follows:
scss
f(x) = max(0, x)
The following is a Python sample code using the ReLU function:
python
import numpy as np
def relu(x):
return np.maximum(0, x)
# Example with a single value
x = 2
result = relu(x)
print(result) # output: 2
# Example using NumPy arrays
x_array = np.array([-2, -1, 0, 1, 2])
result_array = relu(x_array)
print(result_array) # output: [0 0 0 1 2]
In the above example, we defined a relu function that takes an input value x and returns the computed result. Then we used a single value and a NumPy array as an example, calculated the corresponding ReLU function value, and printed the output.
The ReLU function is widely used as an activation function in machine learning and deep learning. Compared with the Sigmoid function and Tanh function, the ReLU function has the following advantages:
Non-linearity: The ReLU function introduces non-linear characteristics, enabling neural networks to learn and represent more complex functional relationships.
Gradient sparsity: On the positive interval, the derivative of the ReLU function is 1, which makes the gradient calculation of backpropagation simpler and more efficient.
Alleviate the problem of gradient disappearance: the ReLU function has no upper bound, and will not cause the problem of gradient saturation and gradient disappearance.
It should be noted that the output of the ReLU function is zero on the negative interval, which may lead to the "death" problem of neurons. To solve this problem, variants of ReLU such as Leaky ReLU and Parametric ReLU can be used.
In practical applications, the ReLU function is often used in hidden layers and convolutional neural networks of deep learning. It helps to improve the expression ability and convergence speed of the model, and it can also reduce the problem of gradient disappearance.