Numerical stability of deep learning models - explanation of gradient decay and gradient explosion

0. Preface

In accordance with international practice, I would like to first declare: This article is only my own understanding of learning. Although I refer to the valuable insights of others, the content may contain inaccuracies. If you find errors in the article, I hope to criticize and correct them so that we can make progress together.

The purpose of this article is to explain common problems regarding numerical stability in deep learning network models: gradient decay (vanishing) and explosion (explosion), as well as common solutions.

Part of the content, opinions and illustrations of this article are based on the lectures of the School of Computer Science at the University of Toronto - Lecture 15: Exploding and Vanishing Gradients , as well as Chapter 3.15 "Numerical Stability and Model Initialization" of Dive into deep learning.

1. Why do gradient decay and gradient explosion occur?

Let’s use the simplified fully connected neuron network below to explain. This fully connected neuron network has only one neuron in each layer and can be regarded as a network connected by a series of neurons.

In forward propagation, since the transmission of numerical values ​​needs to go through the nonlinear activation function $\sigma \left(\right)$ (such as Sigmoid, Tanh function), its numerical size is limited, so前向传播一般不存在数值稳定性的问题.

In backpropagation, for example, solving for the output $y$ pair weight$w_1$的偏导为：
$$\frac{\partial y}{\partial w_1}=p^{\prime }(z_n)w_n\cdot p^{\prime }\left(z_{n-1}\right)w_{n-1}\cdot \cdot \cdot p^{\prime }(z_1)x$$
$$z_n=\{\begin{array}{cc}{w}_n\cdot h_{n-1}+b_n,& n>1\\ w_1\cdot x+b_1,& n=1\end{array}$$
It can be seen here that if the weight $w_n$The initial choice of is unreasonable, or $w_n$During the gradual optimization process, it appears that $p^{\prime }\left(z_n\right)w_n$If most or all are greater than 1 or less than 1, and the network is deep enough, it will cause numerical instability in the partial derivative of back propagation - gradient decay or gradient explosion.

To simplify the understanding, assume $p^{\prime }(z_n)w_n=0.8$ , there are 50 layers of network depth,$0.8^{50}=0.000014$；假设$p^{\prime }\left(z_n\right)w_n=1.2$ , there are 50 layers of network depth,$1.2^{50}=9100$。

Another way to explain numerical stability, referring to Lecture 15: Exploding and Vanishing Gradients, is that deep learning networks are analogous to the iterative use of nonlinear equations, such as $f(x)=3.5x(1-x)$ after multiple iterations$y=f(f(\cdot \cdot \cdot The$ situation$after$$f$$($

_ (Corresponding to gradient explosion).

We should also notice that after 6 iterations $\frac{\partial y}{\partial x}\approx The\; area\; of\; 0$ (corresponding to gradient decay).

2. How to improve numerical stability?

2.1 Random initialization of model parameters

This is the simplest and most commonly used method to combat gradient decay and gradient explosion. As explained above: $p^{\prime }(z_n)w_n$If most or all are greater than 1 or less than 1, and the network is deep enough, numerical instability may easily occur. If you initialize the model parameters randomly, it will largely avoid the problem of $w_n$Unreasonable initial selection leads to gradient decay or explosion.

Xavier random initialization is a commonly used method: assuming that the number of inputs to a hidden layer is $a$ , the number of outputs is$b$ , Xavier random initialization will randomly sample the weight parameters in this layer from$(-\sqrt{\frac{6}{a+b}},\sqrt{\frac{6}{a+b}})$。

2.2 Gradient Clipping

This is a method of artificially limiting the gradient that is too large or too small. The idea is to give the original gradient $g$ plus a coefficient, in$When\; the\; absolute\; value\; of\; g$ is too large, reduce it and vice versa. This coefficient is:
$$\frac{}{||g||}$$

Among them $eta$ is a hyperparameter,$||g||$ is the second norm of the gradient.

Although increasing this coefficient will result in the result not being the true partial derivative of the loss function with respect to the weight, it can maintain numerical stability.

2.3 Regularization

This is a way to suppress exploding gradients. I have introduced the regularization method before: actual weight attenuation based on PyTorch - L2 norm regularization method (with code) . The idea is to add the norm of the weight as a penalty term in the loss function:
$$loss=\frac{1}{n}S(y-\hat{y}{)}^2+\frac{}{2n\_}||w|{|}^{2In}$$
the process of continuous iteration (learning) of the deep learning model,$l\; Asoss$ gets smaller and smaller, the norm of the weight also gets smaller and smaller, which suppresses gradient explosion.

2.4 Batch Normalization

Batch Normalization is a data standardization method that adds scaling and shifting based on Normalization. For its specific working principle, please refer to: Description of Batch Normalization .

The basic principle that Batch Normalization can maintain numerical stability is similar to gradient clipping: both artificially increase the scaling of the numerical value and maintain the numerical value within a reasonable range that is neither too large nor too small. The difference between the two is that gradient clipping directly acts on the partial derivative of the loss function with respect to the weight in the back propagation process; while Batch Normalization standardizes the output of a certain layer in the forward propagation process and indirectly maintains the stability of the partial derivative of the weight. .

What needs to be pointed out here is: due to input $x$ also participates in the calculation of partial derivatives, if$x$ is a high-dimensional vector, then for the input$Batch\; Normalization\; processing\; of\; x$ is also necessary.

2.5 LSTM？Short Cut！

Many articles indicate that LSTM (long short-term memory) networks can help maintain numerical stability. I was puzzled when I first saw these articles - because we need a general method to improve the numerical stability of existing models, and It is not a direct replacement with the LSTM network model. Moreover, LSTM is not a universal deep learning model. It is impossible to replace the model with LSTM when encountering gradient decay or gradient explosion.

If you don’t know what LSTM is, you can read this: Algorithm introduction and mathematical derivation of LSTM (long short-term memory) network

Later, I saw Lecture 15: Exploding and Vanishing Gradients and understood the misunderstanding: this article uses RNN as an example to illustrate numerical stability. For RNN, LSTM is indeed an improved model, because its internal "gate" structure that maintains "long-term memory" really helps improve numerical stability.

I think most articles that single out LSTM to show that it can improve numerical stability have misunderstood.

The structure of Short Cut is a universal rule to improve numerical stability, and LSTM is just a special case of improving RNN.

For the specific mechanism of Short Cut, please refer to He Kaiming’s original text: Deep Residual Learning for Image Recognition