[Hands-on Deep Learning v2 Li Mu] Study Notes 09: Numerical Stability, Model Initialization, Activation Function

Review of the previous article: the discard method

1. Numerical stability

1.1 Introduction

Numerical stability is a very important content in deep learning, especially when the number of network layers is very deep, the numerical value can easily become unstable.

Gradients of Neural Networks

• Consider the following neural network with d layers:
  $${\vec{h}}^t=f_t({\vec{h}}^{t-1})\text{and}y=l\circ f_d\circ \cdots \circ f_1(\vec{x})$$
• Calculate the loss $l$ Regarding the parameter$W_t$Gradient of
  $$\frac{\partial l}{\partial W^t}=\frac{\partial l}{\partial {\vec{h}}^d}\underset{d-t\text{matrix multiplications}}{\underbrace{\frac{\partial {\vec{h}}^d}{\partial {\vec{h}}^{d-1}}\cdots \frac{\partial {\vec{h}}^{t+1}}{\partial {\vec{h}}^t}}}\frac{\partial {\vec{h}}^t}{\partial W^t}$$Since the derivative of a vector with respect to a vector is a matrix, we would do $d-t$ times of matrix multiplication. And too many matrix multiplications can lead to problems with numerical stability.

1.2 Two common problems

• The two most common problems caused by numerical stability are exploding and vanishing gradients , for example:
  $$\begin{gathered} 1.5^{100}\approx 4\times 10^{17} \\ 0.8^{100}\approx 2\times 10^{-10} \end{gathered}$$

• For example: MLP
  Suppose the following MLP (offset omitted for simplicity). No. ttThe output of layer$t$$The\; input\; of\; the\; t$ layer is multiplied by the weight, and then passed through the activation function:
  $$f_t({\vec{h}}^{t-1})=s\left(W^t{\vec{h}}^{t-1}\right)\sigma \text{is the activation}$$ functiontt$t$ layer output pair$The\; derivative\; of\; the\; input\; of\; the\; t$ -layer is first deriving the activation function, and then multiplying by the transposition of the weight matrix:
  $$\frac{\partial {\vec{h}}^t}{\partial {\vec{h}}^{t-1}}=diag(\sigma \_{}^{\prime }(W^t{\vec{h}}^{t-1}))(W^t{)}^T{p}^{\prime }\text{is}\text{the derivative function of}\sigma$$. Therefore, the$The\; derivative\; of\; the\; t$ -layer output to the original input is the multiplicationof a series of derivatives and the transposed matrix:
  $$\prod _{i=t}^{d-1}\frac{\partial {\vec{h}}^{i+1}}{\partial {\vec{h}}^i}=\prod _{i=t}^{d-1}\mathrm{diag}(p^{\prime }(W^i{\vec{h}}^{i-1}))(W^i{)}^T$$

1.3 Gradient Explosion

1.3.1 Causes

• If we use the ReLU function as the activation function.
  $$\sigma \left(x\right)=\max (0,x)\text{and}{p}^{\prime }(x)=\{\begin{array}{l}1\text{if }x>0\\ 0\text{otherwise}\end{array}$$
• ${\prod }_{i=t}^{d-1}\frac{\partial {\vec{h}}^{i+1}}{\partial {\vec{h}}^i}={\prod }_{i=t}^{d-1}\mathrm{diag}(p^{\prime }(W^i{\vec{h}}^{i-1}))(W^i{)}^{Some\; elements\; of\; T}$ will come from${\prod }_{i=t}^{d-1}(W^i{)}^T$
  • If $d-If\; t$ is very large, the value of this multiplication will be very large.

1.3.2 causing problems

• Value out of range (infinity)
  • It is especially serious for 16-bit floating point numbers (value range $(6^{-5},6^4)$）
• sensitive to learning rate
  • If the learning rate is too large, it will result in large parameter values ​​and thus larger gradients.
  • If the learning rate is too small, it will lead to no progress in training.
  • We may need to keep adjusting the learning rate during retraining.

1.4 Gradient disappearance

1.4.1 Causes

• If we use the sigmoid function as the activation function.
  $$\sigma \left(x\right)=\frac{1}{1+e^{-x}}{p}^{\prime }(x)=\sigma \left(x\right)(1-s(x))$$
• When the input is relatively large, the gradient will be small (close to 0).
• ${\prod }_{i=t}^{d-1}\frac{\partial {\vec{h}}^{i+1}}{\partial {\vec{h}}^i}={\prod }_{i=t}^{d-1}\mathrm{diag}(p^{\prime }(W^i{\vec{h}}^{i-1}))(W^i{)}^{The\; element\; values\; of\; T}$ will be$d-The\; product\; of\; t$ decimal values.
  • Like $0.8^{100}\approx 2\times 10^{-10}$

1.4.2 causing problems

• The gradient value becomes 0.
  • This is especially true for 16-bit floating point numbers.
• Training is not progressing.
  • Regardless of how the learning rate is chosen.
• Especially for the bottom layer.
  • Only the top layers are trained better.
  • There is no way to make the neural network deeper.

1.5 Summary

• When the value is too large or too small, it will cause numerical problems.
• Often occurs in deep models because it multiplies n numbers.

2. Make training more stable

2.1 Overview

• Goal: Make the gradient value within a reasonable range, for example: $[1^{-6},1^3]$
• method:
  • Turn multiplication into addition, for example: ResNet, LSTM
  • Normalized
    • Gradient normalization, gradient clipping
  • Reasonable weight initialization and activation functions

This article mainly explains how to properly initialize the weights and use an appropriate activation function .

2.2 Weight initialization

2.2.1 Objective: Let the variance of each layer be a constant.

• Treat the output and gradient of each layer as random variables.
• Keep their mean and variance the same.

Forward	reverse
$$\begin{gathered} \mathbb{E}[h_i^t]=0 \\ Wherer\left[h_i^t\right]=a \end{gathered}$$ $a$ is a constant	$$\mathbb{E}\left[\frac{\partial l}{\partial h_i^t}\right]=0was[\_\frac{\partial l}{\partial h_i^t}]=b\forall i,t$$ $b$ is a constant

2.2.2 Problem Analysis

• Randomize the initial parameters within a range of reasonable values.
• More prone to numerical instabilities at the beginning of training
  • The loss function surface can be complex far from the optimal solution.
  • Surfaces near the optimal solution will be relatively flat.
• Use $\mathcal{N}(0,0.01)$ may be fine for small networks, but not guaranteed for deep neural networks.

What should we do if we want to satisfy the previous goal that both the variance and the mean are constant?
Let's look at an example first: MLP

• suppose
  • $w_{i,j}^t$是$i.i.d$ (independent and identical distribution), then$E[w_{i,j}^t]=0,r[w\_{}_{i,j}^t]=c_t$
  • $h_i^{t-1}$Independent of $w_{i,j}^t$
• Assuming there is no activation function , then ${\vec{h}}^t=W^t{\vec{h}}^{t-1}$，这里 $W^t\in {\mathbb{R}}^{n_t\times n_{t-1}}$
  • Properly, independently : $$\mathbb{E}[h_i^t]=\mathbb{E}\left[\sum _j{w}_{i,j}^t{h}_j^{t-1}\right]=\sum _{j}E[w_{i,j}^t]\mathbb{E}[h_j^{t-1}]=0$$
  • 正向方差：$$\begin{array}{rl}Wherer[h_i^t]& =\mathbb{E}[(h_i^t{)}^2]-(\mathbb{E}[h_i^t]{)}^2=\mathbb{E}\left[\right(\sum _j{w}_{i,j}^t{h}_j^{t-1}{)}^2]\\ & =\mathbb{E}[\sum _j(w_{i,j}^t{)}^2(h_j^{t-1}{)}^2+\underset{\text{Independent and identically distributed, the expectation of this term is 0}}{\underbrace{\sum _{j\ne k}{w}_{i,j}^t{w}_{i,k}^t{h}_j^{t-1}{h}_k^{t-1}}}]\\ & =\sum _{j}E\left[\right(w_{i,j}^t{)}^2\left]\mathbb{E}\right[(h_j^{t-1}{)}^2]\\ & =\sum _{j}r[w\_{}_{i,j}^t]Wherer\left[h_j^{t-1}\right]\\ & =n_{t-1}{c}_{t}Wherer[h_j^{t-1}]\end{array}$$It can be seen that if you want to make the positive variances the same, you need to satisfy: $n_{t-1}{c}_t=1$
  • 反向均值：
    跟正向情况相似$$\frac{\partial l}{\partial {\vec{h}}^{t-1}}=\frac{\partial l}{\partial {\vec{h}}^t}{W}^t\Rightarrow (\frac{\partial l}{\partial {\vec{h}}^{t-1}}{)}^T=(W^t{)}^T(\frac{\partial l}{\partial {\vec{h}}^t}{)}^T$$ $$\mathbb{E}\left[\frac{\partial l}{\partial h_i^{t-1}}\right]=0$$
  • 反向方差：
    $$was[\_\frac{\partial l}{\partial h_i^{t-1}}]=n_t{c}_{t}was[\_\frac{\partial l}{\partial h_j^t}]$$ is similar to the forward case, if you want to make the reverse variances the same, you need to satisfy:$n_t{c}_t=1$
  • It can be solved by using Xavier initialization , see "2.2.3 Xavier initialization" below for details.

2.2.3 Initialization of Xavier

• It is difficult to satisfy $n_{t-1}{c}_t=1$ 和 $n_t{c}_t=1$。
• Xavier使得 $c_t(n_{t-1}+n_t)/2=1\to c_t=2/(n_{t-1}+n_t)$
  • Normal distribution: $\mathcal{N}(0,\sqrt{2/(n_{t-1}+n_t)})$
  • 均匀分布：$\mathcal{U}(-\sqrt{6/(n_{t-1}+n_t)},\sqrt{6/(n_{t-1}+n_t)})$
    • distribution $\mathcal{U}[-a,The\; variance\; of\; a]$ is$a^2/3$
• Adapt weight shape transformations, specifically $n_t$。

2.3 Activation function

2.3.1 Problem Analysis

Undertake the assumptions in "2.2.2 Problem Analysis" above:

• Assuming a linear activation function
  • Suppose $\sigma \left(x\right)=\alpha x+\beta$
    $${\vec{h}}^{\prime }=W^t{\vec{h}}^{t-1}\text{and}{\vec{h}}^t=s({\vec{h}}^{\prime })$$
  • Syntax: $$\mathbb{E}[h_i^t]=E[a{h}_i^{\prime }+b]=\beta This$$ means that if you want to make the expectation 0, you need to satisfy$b=0$
  • Solution: $$\begin{array}{rl}Wherer[h_i^t]& =\mathbb{E}[(h_i^t{)}^2]-(\mathbb{E}[h_i^t]{)}^2\\ & =E[(a{h}_i^{\prime }+b{)}^2]-b^2\\ & =E[a^2(h_i^{\prime }{)}^2+2ab{h}_i^{\prime }+b^2]-b^2\\ & =a^{2}Var[h_i^{\prime }]\end{array}$$If the variances are made the same, it needs to satisfy $a=1$
  • Inverse expectation:
    Suppose $\sigma \left(x\right)=\alpha x+\beta$ $$\frac{\partial l}{\partial {\vec{h}}^{\prime }}=\frac{\partial l}{\partial {\vec{h}}^t}(W^t{)}^T\text{and}\frac{\partial l}{\partial {\vec{h}}^{t-1}}=a\frac{\partial l}{\partial {\vec{h}}^{\prime }}$$ $$\mathbb{E}\left[\frac{\partial l}{\partial h_i^{t-1}}\right]=0$$ If you want to make the expectation 0, you need to satisfy$b=0$
  • 反向方差：
    $$was[\_\frac{\partial l}{\partial h_i^{t-1}}]=a^{2}Vr\left[\frac{\partial l}{\partial h_j^{\prime }}\right]$$ If the variance is the same, it needs to satisfy$a=1$
  • This means that our activation function must be $\sigma \left(x\right)=x$

2.3.2 Check common activation functions

• Using Taylor expansion:
  $$sigmoid(x)=\frac{1}{2}+\frac{x}{4}-\frac{x^3}{48}+O(x^5)$$ $$tenglish\left(x\right)=0+x-\frac{x^3}{3}+O(x^5)$$ $$readagain\left(x\right)\_=0+x\text{ for }x\ge 0$$ We find that for$tanh\left(x\right)$ sum$relu\left(x\right)$ function can be approximated as a linear function y = xy=xnear the zero point$y=x$；而 $sigmoid\left(x\right)$ will not work.
• Therefore, our $sigmoid\left(x\right)$ is adjusted:
  $$4\times sigmoid(x)-2$$ adjusted$sigmoid\left(x\right)$ can also be approximated as$y=x$

2.5 Summary

• Reasonable selection of weight initial values ​​and activation functions can improve numerical stability.