Activation function

What is the activation function?

The activation function is an extremely important feature of artificial neural networks; the activation function determines whether a neuron should be activated, and activation represents that the information received by the neuron is related to the given information; the activation function performs a nonlinear transformation on the input information, and then The transformed output information is passed to the next layer of neurons as input information.

The role of activation function

If an activation function is not used, the output of each layer is a linear function of the input of the upper layer. No matter how many layers the neural network has, the final output is a linear combination of the inputs. The activation function introduces nonlinear factors to neurons, allowing the neural network to arbitrarily approximate any nonlinear function.

Types of activation functions

Identity

identity is suitable for tasks where the underlying behavior is linear (similar to linear regression). However, nonlinear mapping cannot be provided. When a multi-layer network uses the identity activation function, the entire network is equivalent to a single-layer model.

The function is defined as $f(x)=x$ , the derivative is$f^{{}^{\prime }}(x)=1$。

Step

The activation function Step is more theoretical than practical, mimicking the all-or-nothing properties of biological neurons. However, it cannot be applied to neural networks because its derivatives are 0 (except for the zero-point derivative, which is undefined), which means that gradient-based optimization methods are not feasible.

The definition of the function is $f\left(x\right)=\{\begin{array}{l}\begin{array}{cc}0& x<0\end{array}\\ \begin{array}{cc}1& x\ge 0\end{array}\end{array}$，导数为 $f^{{}^{\prime }}(x)=\{\begin{array}{l}\begin{array}{cc}0& x\ne 0\end{array}\\ \begin{array}{cc}?& x=0\end{array}\end{array}$。

Sigmoid

The output mapping of the sigmoid function is between (0,1), monotonic and continuous, the output range is limited, the optimization is stable, and can be used as the output layer; and the derivation is easy. However, due to its soft saturation, once it falls into the saturation zone, the gradient will be close to 0. According to the chain rule of backpropagation, the gradient will easily disappear, causing training problems; the output of the Sigmoid function is always greater than 0. The non-zero centered output will bias the input of the neuron in the subsequent layer (Bias Shift), and further slow down the convergence speed of gradient descent; and during calculation, due to the power operation, the computational complexity Higher, the operation speed is slower.

The function is defined as $f(x)=\sigma \left(x\right)=\frac{1}{1+e^{-x}}$, the derivative is $f^{{}^{\prime }}(x)=\frac{e^{-x}}{(1+e^{-x}{)}^2}=f(x)(1-f(x))$。

Fishy

Tanh converges faster than the sigmoid function. Compared with the sigmoid function, tanh is centered on 0. However, like the sigmoid function, the gradient disappears easily due to saturation. Due to the power operation, the calculation complexity is high and the operation speed is slow.

The function is defined as $f(x)=the\left(x\right)=\frac{e^x-e^{-x}}{e^x+e^{-x}}$, the derivative is $f^{{}^{\prime }}(x)=1-tan{h}^2(x)=1-f(x{)}^2$。

resume

The ReLU function converges quickly. Compared with sigmoid and tanh, which involve power operations, resulting in high computational complexity, ReLU​ can be implemented more simply; when the input x>=0, the derivative of ReLU​ is a constant, which can be effective Alleviates the vanishing gradient problem; when x<0, the gradient of ReLU is always 0, providing the sparse expression capability of the neural network. However, the output of ReLU is not centered on 0; and due to neuron necrosis, some neurons may never be activated, causing the corresponding parameters to never be updated, and the gradient explosion problem cannot be avoided.

The definition of the function is $f\left(x\right)=\{\begin{array}{l}\begin{array}{cc}0& x<0\end{array}\\ \begin{array}{cc}x& x\ge 0\end{array}\end{array}$，导数为 $f^{\prime }(x)=\{\begin{array}{l}\begin{array}{cc}0& x<0\end{array}\\ \begin{array}{cc}1& x\ge 0\end{array}\end{array}$

LReLU

The LReLU activation function can avoid gradient disappearance. Since the derivative is always non-zero, it can reduce the appearance of dead neurons; however, LReLU​ performance is not necessarily better than ReLU​, and the gradient explosion problem cannot be avoided.

The definition of the function is $f(x)=\{\begin{array}{l}\begin{array}{cc}\alpha x& x<0\end{array}\\ \begin{array}{cc}x& x\ge 0\end{array}\end{array}$，导数为 ${f(x)}^{{}^{\prime }}=\{\begin{array}{l}\begin{array}{cc}& x<0\end{array}\\ \begin{array}{cc}1& x\ge 0\end{array}\end{array}$。

PRELU

PReLU​ is an improvement of LReLU. It can adaptively learn parameters from data, has fast convergence speed and low error rate. PReLU can be used for backpropagation training and can be optimized simultaneously with other layers.

The definition of the function is $f(a,x)=\{\begin{array}{l}\begin{array}{cc}\alpha x& x<0\end{array}\\ \begin{array}{cc}x& x\ge 0\end{array}\end{array}$, the derivative of the function is $f^{\prime }(a,x)=\{\begin{array}{l}\begin{array}{cc}& x<0\end{array}\\ \begin{array}{cc}1& x\ge 0\end{array}\end{array}$。

RReLU

RReLU adds a linear term for negative inputs, and the slope of this linear term is randomly assigned at each node (usually uniformly distributed).

The definition of the function is $f(a,x)=\{\begin{array}{l}\begin{array}{cc}\alpha x& x<0\end{array}\\ \begin{array}{cc}x& x\ge 0\end{array}\end{array}$，导数为 ${f(a,x)}^{{}^{\prime }}=\{\begin{array}{l}\begin{array}{cc}& x<0\end{array}\\ \begin{array}{cc}1& x\ge 0\end{array}\end{array}$。

UP

The ELU derivative converges to zero, thereby improving learning efficiency. Using ELU can get negative outputs, which can help the network push weight and bias changes in the right direction, and also prevent dead neurons from appearing. However, due to the large amount of calculation, its performance is not necessarily better than ReLU, and the gradient explosion problem cannot be avoided.

函数的定义为 $f(a,x)=\{\begin{array}{l}\begin{array}{cc}a\left(e^x-1\right)& x<0\end{array}\\ \begin{array}{cc}x& x\ge 0\end{array}\end{array}$，其导数为 ${f^{\prime }(a,x)}^{}=\{\begin{array}{l}\begin{array}{cc}f(a,x)+& x<0\end{array}\\ \begin{array}{cc}1& x\ge 0\end{array}\end{array}$。

THE VILLAGE

SELU is a variant of ELU. After passing through this activation function, the sample distribution is automatically normalized to 0 mean and unit variance, and there will be no gradient disappearance or explosion problems.

Independent of $f(a,x)=l\{\begin{array}{l}\begin{array}{cc}a\left(e^x-1\right)& x<0\end{array}\\ \begin{array}{cc}x& x\ge 0\end{array}\end{array}$Let ${f^{\prime }(a,x)}^{}=l\{\begin{array}{l}\begin{array}{cc}a\left(e^x\right)& x<0\end{array}\\ \begin{array}{cc}1& x\ge 0\end{array}\end{array}$. where $l$和$\alpha$ is a fixed value (1.0507 and 1.6726 respectively).

Softsign

softsign is another alternative to the tanh activation function. softsign is antisymmetric, decentered, differentiable, and returns a value between −1 and 1, and softsign's flatter curve and slower descending derivative indicate that it can be more efficient. to learn. But the calculation of the derivative is more troublesome than tanh.

The function is defined as $f(x)=\frac{x}{\left|x\right|+1}$, the derivative is $f^{{}^{\prime }}(x)=\frac{1}{{(1+\left|x\right|)}^2}$。

Softplus

softplus is a good alternative to relu, capable of returning any value greater than 0. Unlike relu, the derivative of softplus is continuous, non-zero, and everywhere, thus preventing dead neurons. However, the derivative is often less than 1, and the problem of vanishing gradient may also occur. Another difference between softplus and relu is its asymmetry, which is not centered on zero, which may hinder learning.

The definition of the function is $f\left(x\right)=\ln (1+e^x)$, the derivative of the function is$f^{{}^{\prime }}\left(x\right)=\frac{1}{1+e^{-x}}$。

Softmax

The softmax function is generally used in multi-classification problems. It is an extension of logistic regression and is also called multi-nominal logistic regression model (multi-nominal logistic mode). Suppose you want to implement $For\; the\; classification\; task\; of\; k$ categories, the Softmax function will input data$x_i$mapped to iiProbability yi y_i$of\; i$ categories$y_i$ 如下计算：$$y_i=soft\max \left(x_i\right)=\frac{e^{x_i}}{{\sum }_{j=1}^k{e}^{x_j}}$$显然，$0<y_i<1$。

Swish

For the Swish function, when x>0, there is no gradient disappearance; when x<0, neurons will not die like ReLU. swish is differentiable everywhere, continuously smooth, and swish is not a monotonic function. Using swish can improve the performance of the model, but swish is computationally intensive.

The function is defined as $f\left(x\right)=x\cdot p\left(x\right)$，导数为$\begin{array}{c}{f}^{{}^{\prime }}\left(x\right)=f\left(x\right)+p\left(x\right)\left(1-f\left(x\right)\right)\end{array}$. where $\sigma \left(x\right)$ is the sigmoid function.

HSwish

Compared with swish, hard swish reduces the amount of calculation and has the same properties as swish. However, the calculation amount of hard swish is still larger than that of relu.

函数的定义为 $f\left(x\right)=x\frac{\text{RELU}6\left(x+3\right)}{6}=x\cdot \{\begin{array}{ll}1,& x\ge 3\\ \frac{x}{6}+\frac{1}{2},& -3<x<3\\ 0,& x\le -3\end{array}$，导数为 $f^{\prime }(x)=\{\begin{array}{ll}1,& x\ge 3\\ \frac{x}{3}+\frac{1}{2},& -3<x<3\\ 0,& x\le -3\end{array}$。

Choice of activation function

1. When using shallow networks as classifiers, sigmoid functions and their combinations usually work better.

2. Due to the vanishing gradient problem, it is sometimes necessary to avoid using the sigmoid and tanh functions.

3. The relu function is a general activation function and is currently used in most cases.

4. If dead neurons appear in the neural network, then the prelu function is the best choice.

5. The relu function can only be used in hidden layers.

6. Usually, you can start with the relu function. If the relu function does not provide optimal results, try other activation functions.

Activation function related issues

1. Why can relu be used for gradient-based learning even if it is not fully differentiable/differentiable?

From a mathematical point of view, relu is not differentiable at 0 because its left derivative and right derivative are not equal; but when implemented, one of the left derivative or the right derivative is usually returned instead of reporting an error that the derivative does not exist, thus This problem is avoided.

2. Why does tanh converge faster than sigmoid?

$$\tan {\text{h}}^{{}^{\prime }}\left(x\right)=1-\tan \text{h}{\left(x\right)}^2\in \left(0,1\right)$$$$s^{{}^{\prime }}\left(x\right)=s\left(x\right)\left(1-s\left(x\right)\right)\in \left(0,\frac{1}{4}\right]$$ From the above two formulas, it can be seen that the gradient disappearance problem caused by tanh is not as serious as sigmoid, so the convergence speed of tanh is faster than that of sigmoid.

3. What is the difference between sigmoid and softmax?

• For binary classification problems, sigmoid and softmax are the same. They both seek cross entropy loss, while softmax can be used for multi-classification problems.

• Softmax is an extension of sigmoid, because when the number of categories k=2, softmax regression degenerates into logistic regression.

• The distribution used in softmax modeling is a polynomial distribution, while logistic is based on the Bernoulli distribution.

• Multiple logistic regressions can also achieve multi-classification effects through superposition, but softmax regression performs multi-classification. Classes are mutually exclusive, that is, one input can only be classified into one class; multi-logistic regression performs multi-classification. , the output categories are not mutually exclusive, that is, the word "apple" belongs to both the "fruit" category and the "3C" category.