Nonlinear activation functions are an important part of deep learning networks. With the rapid development in recent years, more and more activation functions have been proposed and improved. Choosing an appropriate activation function will determine the final result of the model. The following summarizes the calculation methods and corresponding images of 13 common activation functions. The calculation methods in this article come from pytorch.
1.Sigmoid
This is an earlier activation function, and its calculation formula is as follows:
Its diagram is as follows:
advantage:
The output of the Sigmoid function is between (0,1), the output range is limited, the optimization is stable, and it can be used as the output layer.
Continuous function, easy to find derivatives.
shortcoming:
Requires exponentiation and is computationally expensive.
The output is not averaged at 0, resulting in a decrease in the convergence rate.
Gradient dispersion is prone to occur. During backpropagation, when the gradient is close to 0, the weights will basically not be updated, so that the training of the deep network cannot be completed.
2.LogSigmoid
Its calculation formula is as follows:
Its diagram is as follows:
3. Replay
One of the most commonly used activation functions in deep neural networks, its calculation formula is as follows:
Its diagram is as follows:
advantage:
In the area of x>0, there will be no problems of gradient saturation and gradient disappearance, and the convergence speed is fast.
No exponent operation is required, so the operation speed is fast and the complexity is low.
shortcoming:
The mean of the output is non-zero.
There is neuron death, and at x<0, the gradient is 0. The gradient of this neuron and subsequent neurons is always 0, and no longer responds to any data, resulting in the corresponding parameters never being updated.
4. LeakyRelu
The above Relu outputs 0 when x is less than 0, while LeakyRelu can output a non-zero value when x is less than 0, and its calculation formula is as follows:
Its diagram is as follows:
advantage:
Solve Relu's neuron death problem, have a small positive slope in the negative region, so it can backpropagate even for negative input values.
Has the advantage of the Relu function.
shortcoming:
Inconsistent results, unable to provide consistent relationship predictions for positive and negative input values
5. High
It is also an improvement for the negative part of ReLU. The ELU activation function uses an exponential calculation method to output the case where x is less than zero. The calculation formula is as follows:
Its diagram is as follows:
advantage:
is continuously differentiable at all points.
Faster training time compared to other linear non-saturated activation functions such as Relu and its variants.
There is no question of neuron death.
As a non-saturated activation function, it does not suffer from exploding or vanishing gradients and has higher accuracy.
shortcoming:
Exponentiation is involved, and the calculation speed is slow.
6. Prelude
Among them, a is not fixed, but is learned through backpropagation. Its calculation formula is as follows:
Its diagram is as follows:
7.Relu6
Relu6 limits the output of Relu to no more than 6, and its calculation formula is as follows:
Its diagram is as follows:
Relu uses x for linear activation in the area of x>0, which may cause the activated value to be too large and affect the stability of the model. In order to offset the linear growth part of the ReLU activation function, the Relu6 function can be used.
8. RRelu
"Random Rectified Linear Unit" RRelu is also a variant of LeakyRelu. In RRelu, the slope of negative values is random during training and becomes fixed during subsequent testing. Its calculation formula is as follows:
Its diagram is as follows:
9. SELU
The forms of SElu and Elu are similar, but there is one more scale parameter. Its calculation formula is as follows:
Its diagram is as follows:
acale=1.0507009873554804934193349852946 in pytorch.
10. Purpose
Similar to the above SElu, CElu also uses the negative range as an exponential calculation, and the integer range as a linear calculation. The calculation formula is as follows:
Its diagram is as follows:
11. GElu
The regularization method is added to the activation function, and its calculation formula is as follows:
Because erf has no analytical expression, the original paper gives an approximate solution.
Its diagram is as follows:
12.Tang
The hyperbolic tangent function Tanh in mathematics is also a commonly used activation function for neural networks, especially for the last layer of image generation tasks. Its calculation formula is as follows:
Its diagram is as follows:
advantage:
The output mean value is 0, which makes its convergence speed faster than sigmoid, which can reduce the number of iterations.
shortcoming:
Its disadvantage is that exponentiation is required and the calculation cost is high
There is also gradient disappearance, because there are cases where it approaches 0 on both sides.
13.Tanhshrink
Tanhshrink directly uses the input minus the hyperbolic tangent value, and its calculation formula is as follows:
Its diagram is as follows:
Summarize:
First use ReLU, which is the fastest, and then observe the performance of the model.
If the ReLU effect is not very good, you can try variants such as LeakyRelu
In a CNN whose depth is not particularly deep, the influence of the activation function is generally not too great.