[Miscellaneous notes] Summary of basic optimization algorithms for deep learning

0.

In the training process of deep learning, the three main problems that are easy to encounter are local minimum, saddle point and gradient disappearance.

Local minimum : If the training process falls into a local minimum point, it is very likely that the final result will be a local optimum rather than a global optimum. Of course, if there is a certain degree of noise, it may be possible to jump out of this local minimum. This is also the advantage of mini-batch stochastic gradient descent, i.e. it can move parameters away from local minima with variations on the mini-batch gradient.

Gradient disappearance : There are many ways to alleviate gradient disappearance, such as using ReLU activation function, BatchNorm, etc.
  \space 

1. Gradient descent, stochastic gradient descent, mini-batch gradient stochastic descent

The first thing to note is that most deep learning problems are non-convex. For the loss function $l$ , which has a Heissan matrix:
$$H=[\frac{\partial l}{\partial x_i\partial x_j}{]}_{n\times n}$$
where $n$ is the dimension of the variable (is a vector).
If all the eigenvalues ​​of the Heissan matrix are negative, it has a local maximum; all the eigenvalues ​​​​are positive and negative. But for$When\; n$ is very large, it is difficult to ensure that all eigenvalues ​​have the same sign, which isa saddle point. For a convex function, the eigenvalue of its Heissan matrix must be non-negative, so the optimization problem of deep learning is not convex optimization.

Supplement: Convex function:
definition (convex function):
  \space 
Given a convex set $X$ , function$f:X\to R$ (note: convex functions defined on convex sets) is said to be convex if:
$\lambda f(x_1)+(1-\lambda )f\left(x_2\right)\ge f(\lambda x_1+(1-l)x_2),\forall x_1\le x_2\in X,l\in [0,1]$
  \space  
Convex functions have the following properties :
1. Jensen's inequality (the function whose mean value is greater than or equal to the mean value)
2. The local minimum value is the global minimum value (proof: proof by contradiction)
3. The level set is still a convex set, namely:
S ≜ { x ∣ x ∈ X , f ( x ) ≤ b } S \triangleq\{x|x\in X ,f(x)\le b\}S≜{ x∣x∈X,f(x)≤b } is still convex. (Proof:$\lambda x+(1-l)x^{\prime }\in S,\forall x,x^{\prime }\in S,l\in [0,1],That\; is\; tosayf(\lambda x+(1-l)x^{\prime })\le b$ )
4. As mentioned above, the Heissan matrix is ​​positive semi-definite.
5. Can handle constraint problems:
$$\begin{gathered} \min f(x) \\ s.t.c_i(x)\le 0,i\in The\; method\; of\; I \end{gathered}$$
processing has: 1. Lagrangian multiplier method, 2. Penalty term, 3. Projection (should be to take the projection constraint range. The definition of projection is: the projection of the element outside the collection to the collection is equal to the collection from the collection The closest element to the outer element:$P_X(x)\triangleq \underset{x^{\prime }\in X}{\mathrm{argmin}}||x-x^{\prime }||$）

1.1 Gradient Descent

Intuitively, we should be able to approach the optimal point by updating the parameters in the opposite direction of the gradient direction, that is:
$$x\leftarrow x-\eta \nabla f(\mathbf{x}),wheref:{\mathbb{R}}^d\to R$$

note $x$ represents the parameter. Among them$\eta$ is the learning rate. If it is small, the convergence will be slow. If it is too large, it may oscillate or even diverge.

Newton's method can update the learning rate, but it is not applicable due to excessive storage consumption, which will be explained in detail below.

$fTaylor$ 's equation (infinitesimal):
$$f(x+)\_=f(x)+ϵ\nabla f(x)+\frac{1}{2}{ϵ}^T\mathbf{H}ϵ+o(||ϵ|{|}^3)$$

Both sides pair $Then$ , given:
$$0=\nabla f(x)+\frac{1}{2}(\mathbf{H}+{\mathbf{H}}^T)ϵ=\nabla f\left(x\right)+Hϵ$$
(using$\frac{d(x^{T}Ax)}{dx}=(A^T+A)x$ and the symmetry properties of the Heissan matrix)

Then: $ϵ=-{\mathbf{H}}^{-1}\nabla f(x)$

Obviously to update in this way, you can indeed get closer to the optimal point better, but it takes $O(d^2)$Space complexity to store the Heissan matrix. And if the Heissan matrix has negative eigenvalues ​​for non-convex functions, it may be updated in the opposite direction.

1.2 Stochastic Gradient Descent SGD

Note that the above gradient descent method takes the loss function of the entire sample as input, if there are a total of $n$ samples, the time complexity of calculation is$O\left(n\right)$ , very large.

If we randomly take one sample at a time $i$ , put the sample in parameterx \bm xThe loss function under$x$$f_i\left(x\right)$ , then we use$f_i\left(x\right)$ to estimate the gradient of the entire sample set, which is updated as follows:

$$\mathbf{x}\leftarrow x-\eta \nabla f_i(\mathbf{x})$$

In addition, if the overall loss function is equal to the average of each sample loss function, that is, $f(\mathbf{x})=\frac{1}{n}{\sum }_{i=1}^n{f}_i\left(x\right)$ , then$\nabla f_i\left(x\right)is$ an unbiased estimator of$\nabla$$f$$($$x$$)$
$E[\nabla f_i(\mathbf{x})]=\frac{1}{n}{\sum }_{i=1}^n{f}_i\left(x\right)=\nabla f(\mathbf{x})$

The main disadvantage of SGD is that it cannot use the parallel computing of hardware, and because only the gradient information of one sample is used, the convergence performance is not good.

1.3 Small batch SGD

In order to be able to make good use of the parallel computing characteristics of the GPU, it is a bit wasteful for SGD to take one sample calculation at a time. To this end, we can take the gradient of a batch of samples at a time to replace the original gradient of one sample:

$$\mathbf{x}\leftarrow x-\frac{}{|I_t|}\sum _{i\in I_t}\nabla f_i(\mathbf{x})$$

其中$I_t\subset I$.

2. Momentum method

The gradient is the direction in which the value of the function changes the fastest, but sometimes it causes jitter. If we consider the gradient of the past time (that is, consider the past update direction) when updating the weight, it is like the "inertia" in physics. Going too far all of a sudden.

Consider the previous small batch SGD, remember
$${\mathbf{g}}_{t,t-1}=\frac{1}{|I_t|}\sum _{i\in I_t}\nabla f_i({\mathbf{x}}_{t-1})$$

For convenience, time information is added to the subscript. ${\mathbf{g}}_{t,t-1}$means from $t-1$ moment tottUpdate at time$t$${\mathbf{x}}_{t-1}$is the parameter at the previous moment.

We define momentum:
$$\begin{gathered} {\mathbf{v}}_t=b{v}_{t-1}+g_{t,t-1} \\ {v}_0=0 \end{gathered}$$

That is, the momentum at this moment is represented by the momentum and gradient direction at the previous moment. Obviously, the influence of the gradient at the old moment on the momentum will become smaller and smaller as time goes on. To demonstrate this, it can be expanded: vt
$$\begin{gathered} {\mathbf{v}}_t=b{v}_{t-1}+g_{t,t-1}=v_t=b(b{v}_{t-2}+g_{t-1,t-2})+g_{t,t-1} \\ =...=b^{t-1}(\beta {\mathbf{v}}_0+g_{1,0})+b^0{Mr}_{t,t-1}+\beta g_{t-1,t-2}+...b^{t-2}{g}_{2,1} \\ =\sum _{\tau =0}^{t-1}{b}^t{g}_{t-\tau ,t-\tau -} \end{gathered}$$

Therefore, the gradient information is replaced by momentum information, and the weights are updated as follows:

$$\begin{gathered} {\mathbf{v}}_t\leftarrow b{v}_{t-1}+{\mathbf{g}}_{t,t-1} \\ {\mathbf{x}}_t\leftarrow x_{t-1}-\eta {\mathbf{v}}_t \end{gathered}$$

$b=0$ degenerates into regular small-batch SGD.

3.RMS plug

The AdaGrad algorithm updates the state variable as follows:
$${\mathbf{s}}_t=s_{t-1}+g_t^2$$

$g_t$The meaning of is the same as before. After that, the weights are updated according to the formula:
$$x_t\leftarrow x_{t-1}-\frac{}{\sqrt{s_t+ϵ}}\ast g_t$$

where * stands for element-wise multiplication. Doing so means: for each parameter have a different learning rate (learning rate $\frac{}{\sqrt{s_t+ϵ}}$Different values ​​for each element of the vector)

But as time grows, it is likely ${\mathbf{s}}_t$will continue to grow. To constrain this, consider:

$$s_t=\lambda {\mathbf{s}}_{t-1}+(1-l)g_t^2$$

This is the only difference between RMSprop and AdaGrad.

4.Adam

Combining the previous ideas, the Adam algorithm is produced.

Consider both momentum and state variables:

$$\begin{gathered} {\mathbf{v}}_t=b_1{v}_{t-1}+(1-b_1){\mathbf{g}}_t \\ {s}_t=b_2{s}_{t-1}+(1-b_2)g_t^2 \end{gathered}$$

General1 $1>b_2>b_1$, which means that the update of the state variable is slower than the momentum.

But initializing to 0 value as before will bring a relatively large deviation. Because although there is a gradient at the beginning, but $v_t$The value of etc. will be relatively small.
In order to cope with $In\; the\; case\; where\; the\; gradient\; is\; small\; when\; t$ is small,it can be normalized as follows:

$$\widehat{{\mathbf{v}}_t}=\frac{v_t}{1-b_1^t},\widehat{{\mathbf{s}}_t}=\frac{s_t}{1-b_2^t}$$

The denominator is less than 1, the value can be expanded at the beginning.

Combined with the method of RMSprop, different learning rates are assigned to each parameter:

$${\mathbf{g}}_t^{\prime }=\frac{}{\sqrt{\widehat{s_t}}+ϵ}\ast \widehat{v_t}$$

Practice shows that $\sqrt{\widehat{s_t}}+ϵ$ is more effective than$\sqrt{\widehat{s_t}+ϵ}$good.

Finally update as follows:

$${\mathbf{x}}_t\leftarrow x_{t-1}-g_t^{\prime }$$

Adam is very insensitive to the learning rate.