Summary of various optimizers for deep learning

1. Design principle of optimization algorithm

The principle of the optimization algorithm in deep learning is the gradient descent method, which minimizes the objective function $J\left(\theta \right)$ , the optimal solution process, first solve the gradient of the objective function$\nabla J\left(\theta \right)$ , then the parameter$If\; \theta$ is an independent variable,$i_t=i_{t-1}-\eta \nabla J\left(\theta \right)$，$\eta$ is the learning rate, indicating the size of the gradient update step. The algorithm that the optimization process depends on is called an optimizer. It can be seen that the two cores of the deep learning optimizer are the gradient and the learning rate. The former determines the direction of parameter update, and the latter The latter determines the degree of updating of the parameters.

We define $\theta$ is the parameter to be optimized,$J\left(\theta \right)$ is the objective function, and the initial learning rate is$n$ . The execution framework of the optimization algorithm during the gradient descent process is as follows:

1. Calculate the gradient of the objective function with respect to the current parameters:

$$g_t=\nabla J(i_t)$$

2. Calculate the first-order and second-order momentum of the historical gradient as needed:

$$m_t=\varphi (g_1,g_2,\cdot \cdot \cdot ,g_t)$$

$$V_t=\psi (g_1,g_2,\cdot \cdot \cdot ,g_t)$$

3. Calculate the descending gradient at the current moment:

$$p=the\ast m_t/\sqrt{V_t}(adaptive\; optimizer)$$

$$p=the\ast g_t(non-adaptive\; optimizer)$$

4. Perform gradient descent update

$$i_{t+1}=i_t-p$$

For various optimizers, step $3$ and step$4$ are the same, the main difference is reflected in step$1$ and step$2.$ Let's explain the design ideas of these optimizers in detail.

1. Non-adaptive optimizer

During the optimization process, the learning rate remains unchanged throughout the whole process, or according to a certain $learning$ $schedule$ changes over time, called a non-adaptive optimizer, this category includes the most common$SGD$ (Stochastic Gradient Descent) with$Momentum$ 's$SGD$、带$Nesterov$的$SGD$ et al.

1、$BGD$($Batch$ $gradient$ $descent$)

Its gradient update formula is:

$$i=i-the\ast {\nabla }_{}J(\theta )$$

$SGD$ uses the data of the entire training set to calculate loss lossevery time the gradient is updated$l\; The\; gradient\; ofoss$ to parameters, so it is very slow to calculate, and it will be very tricky when encountering a large number of data sets, and it is not possible to update the model in real time with new data.

2、$SGD$ stochastic gradient descent method ($Stochastic$ $gradient$ $descent$)

Its gradient update formula and $SGD$ is similar.

$The\; gradient\; descent\; process\; of\; SGD$ is similar to a small ball rolling down a hill. Its forward direction is only consistent with the maximum slope direction of the current hill (maximum negative gradient direction), and the initial velocity at each moment is 0. The gradient update rule is: each update for a$Gradient\; updates\; are\; performed\; for\; each\; sample\; inbatch\; ,\; which\; makesthe$ network update parameters very fast. But its shortcomings are also obvious, because$SGD$ updates the gradient very frequently, which will cause$cost$ $Function$ will have serious shocks, and it may oscillate back and forth at the global minimum and jump out of the global optimum.

3、$MSGD$($Mini-batch$ $gradient$ $descent$)

In fact, we more often refer to $MSGD$ calls it$SGD$ , its gradient update formula and$SGD$ is similar. The gradient update rule is:$MSGD$ uses a batch batcheach time$batch$ sample, ie$N$ samples are calculated, so that it can reduce the variance when updating parameters, and the convergence is more stable. On the other hand, it can make full use of the highly optimized matrix operation in the deep learning library for more effective gradient calculation. and$2$ medium$The\; difference\; of\; SGD$ is that instead of using all samples in the data set each time the gradient is updated, it is abatch batchnninside$ba$$t$$c$$h$$n$ samples. SGD SGDmentioned in the article below$SGD$ refers to$MSGD.$ _

advantage:

1. Although it looks like $MSGD\text{'}$ s$cost$ $funct\; io\; nfluctuatesgreatly$ and will take many detours, but the requirements for gradients are very low (calculating gradients is fast), and for the introduction of noise, a lot of theoretical and practical work has proved that as long as the noise is not particularly large,$MSGD$ can converge well.

2. When applying large data sets, the training speed is very fast. For example, take hundreds of data points from millions of data samples each time, and calculate a $MSGD$ gradient, update the model parameters. Compared with traversing all samples of the standard gradient descent method, it is much faster to update the parameters every time a sample is input.

shortcoming:

1. Good convergence cannot be guaranteed; $mini$-$batch$ only uses a part of the data set for gradient descent each time, so each descent is not strictly in the direction of the minimum, but the overall downward trend is in the direction of the minimum, and it is extremely easy to fall into a local minimum.

2、$l\; Ifr$ is too small, the convergence speed will be very slow, if it is too large,$loss$ $Functionwillkeeposcillatingor$ even deviate from the$minimum$$value$$.$(One measure is to set a larger learning rate first, and when the change between two iterations is lower than a certain threshold, reduce$learning$ $rate$ , but the setting of this threshold needs to be written in advance, but in this case it cannot adapt to the characteristics of the data set).

3. For non-convex functions, it is also necessary to avoid being trapped in a local minimum or a saddle point, because the gradient around the saddle point is close to $0$，$MSGD$ could easily get stuck here. To explain the saddle point here is: the curve, surface, or hypersurface of the saddle point neighborhood of a smooth function is located on different sides of the tangent of this point. As shown below$2$ .

4、$SGDM$ (with$Momentum$ SGD SGDof momentum$SGD$)

In order to solve the problems that are prone to occur in the above optimization algorithms, $SGDM$ application was born, in the originalSGD SGDThe first-order momentum is added to the SG D. From an intuitive understanding, it is to add an inertia. In places with steep slopes, there will be greater inertia, and the descent will be$faster$$;$ will be slower.

Its gradient update formula is:

$$v_t=v{\_}_{t-1}+\eta {\nabla }_{}J(\theta )$$

$$i=i-v_t$$

$SGDM$ by adding$v{\_}_{t-1}$, which is defined as momentum, $\gamma$ commonly used value is$0.9$ . After using the momentum, in the gradient descent process, the descent speed can be made faster in the direction where the gradient direction is unchanged, and the descent speed in the direction where the gradient direction is changed is slower. Here we give a closer example: in the original gradient descent, we have been descending in one direction, and when we encounter a ravine (that is, an inflection point), we are on the other side of the hillside when we cross the ravine. At this time The gradient direction is opposite to the previous one. At this time, due to the accumulation of the previous gradient size, the changes between the two hillsides will be canceled out, so it will not always oscillate on the two hillsides, and it is easy to go down the valley. It just reduces vibration. SGDM SGDMThe status of the gradient update of$SG$$D$$M$$3$ , and Figure$1$ , the vibration is significantly reduced.

5.$NAG$ ($Nesterov$ $accelerated$ $gradient$)

SGDM introduced above SGDM$Each\; descending\; step\; of\; SGDM$ is composed of an accumulation of the previous descending direction and the gradient direction of the current point, but when it just descends to the vicinity of the inflection point, if we continue to update the parameters in this way at this time, we will have a A larger magnitude crosses the inflection point, that is, the model will not automatically reduce the magnitude of the update when it encounters an inflection point. NAG improves the momentum method for the above problems, and its expression is as follows:

$$v_t=v{\_}_{t-1}+\eta {\nabla }_{}J(\theta -v{\_}_{t-1})$$

$$i=i-v_t$$

$NAG$ uses the previous gradient value at the current position to perform a parameter update first, and then calculates the gradient at the updated position, and adds this part of the gradient to the previously accumulated gradient value vector. Simply put, it is based on the previous accumulation. The gradient direction of simulates the value after the parameter update in the next step, and then replaces the gradient at the current position in the momentum method with the gradient at the simulated position. So why does this method solve the problem mentioned before? Because$NAG$ has a step to predict the position gradient of the next step, so when it falls near the inflection point,$NAG$ predicts that it will cross the inflection point, and the gradient of this item will be corrected for the previous gradient, which is equivalent to preventing its span from being too large.

As above picture$4$ , we give an example to briefly showNAG NAGThe update principle of$N$$A$$G$$SGDM$ method, the short blue line represents the gradient update of the current position, and the long blue line represents the gradient accumulated before; the first red line represents the use of$TheNAG$ algorithm predicts the gradient update of the next position. The first brown line represents the previously accumulated gradient, and the vector addition result (green line) is the direction of the parameter update$.$$The\; state\; diagram\; of\; NAG$ when updating the gradient is shown in Figure$5$ , it can be seen from the figure that compared to$SGDM$ and$SGD$ ,$NAG$ showed better performance.

2. Adaptive optimizer

In the optimization process, the learning rate changes adaptively with the gradient, and try to eliminate the influence of the given global learning rate as much as possible. It is called an adaptive optimizer, and the common ones include A dagrad $Adagrad$、$Adadelta、RMS$ propRMSprop$\_$$\_$$\_$$\_$$RMSprop$、$Adam$等。
1、$Thedagrad$

$Adagrad$ is actually a constraint on the learning rate. For parameters that are frequently updated, we have accumulated a lot of knowledge about it. We don’t want to be affected too much by a single sample, and we hope that the learning rate will be slower;$for$$occasional$ For the updated parameters, we know too little information, and hope to learn more from each occasional sample, that is, the learning rate is larger. The use of second-order momentum in this method means the arrival of the era of "adaptive learning rate" optimization algorithm.

Here we illustrate the second order momentum $V_t$Definition of : It is used to measure the historical update frequency of parameters, and the second-order momentum is the sum of squares of all gradient values ​​so far. $The\; expression\; of\; Adagrad$ is:

$$m_t=g_t$$

$$V_t=\sum _{i=1}^t{g_t}^2$$

$$i_{t+1}=i_t-the\frac{m_t}{\sqrt{V_t}}$$

where $g_t$for $The\; parameter\; gradient\; at\; time\; t$ , let's explain why$adagrad$ can change the learning rate of its parameters for different frequency$features$$.$First, we see that the second-order momentum$V_t$, which is the cumulative sum of squared gradients. For features with less training data, the corresponding parameters are updated slowly, that is to say, the cumulative sum of squared gradients of their gradient changes will be relatively small, so the learning rate corresponding to the above parameter update equation is will become larger, so for a certain characteristic data set, the corresponding parameter update speed will be faster. In case the above denominator is $0$ , so often add a smoothing term parameter$ϵ$ , the parameter update equation becomes:

$$i_{t+1}=i_t-the\frac{m_t}{\sqrt{V_t+ϵ}}$$

but $adagrad$ also has a problem, that is, its denominator will increase as the number of training increases, which will cause the learning rate to become smaller and smaller, and finally infinitely close to$0$$0$ , making it impossible to effectively update the parameters.

2、$Adadelta$

For A dagrad AdagradCons of$A$$d$$a$$g$$r$$a$$d$$Adadelta$ to the second order momentum$V_t$Improved, and $Compared\; with\; Adagrad$ , the denominator is replaced by the attenuation average value of the gradient square in the past. This denominator is equivalent to the root mean square value RMS RMS of$the$$RMS$($root$ $mean$ $squared)$ . Its expression is as follows:

$$m_t=g_t$$

$$V_{g,t}=\gamma V_{g,t-1}+(1-c){g_t}^2$$

$$V_{Di,}=\gamma V_{D\theta ,t-}+(1-c){D{i}_t}^2$$

$$RMS[g]t=\sqrt{Vg,t+ϵ}$$

$$RMS[\Delta \theta ]t=\sqrt{VDi,t+ϵ}$$

$$i_{t+1}=i_t-\frac{RMS[\Delta \theta {]}_{t-1}}{RMS[g{]}_t}{m}_t$$

where the second-order momentum change to the gradient is $RMS[g{]}_t$; The second-order momentum of the change in the variable is $RMS[\Delta \theta {]}_t$, and replace it with the learning rate using . by $adaadellta$ optimization algorithm, we don't even need to set a default learning rate, because this has been removed in the new rules$.$

3、$RMSprop$

$RMSprop$和$Adadelta$ is all about solving$TheAdagradlearningrate$ drops sharply because of$the$$problem$$.$$RMSprop$与$The\; calculation\; formula\; of\; Adadelta$ is very similar, but it is proposed independently at the same time, and its expression is:

$$m_t=g_t$$

$$V_t=\gamma V_{t-1}+(1-c){g_t}^2$$

$$i_{t+1}=i_t-the\frac{m_t}{\sqrt{V_t+ϵ}}$$

It can be seen from this that $RMSprop$ also needs to manually set an initial learning rate$n$ . The authors suggest that$ϵ$ is set to 0.9, learning rate$\eta$ was set to 0.001.

4、$Adam$($Adaptive$ $Moment$ $Estimation$)

$TheAdamalgorithm$ is another way to calculate an adaptive learning rate for each parameter. It is a kind of momentum$Momentum$ andRMS prop RMSpropThe method of combining$RMSp$$ro$$p$$b_1$and $b_2$, whose expression is:

$$m_t=b_1{m}_{t-1}+(1-b_1)g_t(first\; order\; momentum)$$

$$V_t=b_2{V}_{t-1}+(1-b_2){g_t}^2(Second\; Momentum)$$

$${\tilde{m}}_t=\frac{m_t}{1-b_1^t}$$

$${\tilde{V}}_t=\frac{V_t}{1-b_2^t}$$

$$i_{t+1}=i-the\frac{{\tilde{m}}_t}{\sqrt{{\tilde{V}}_t+ϵ}}$$

Among them $b_1$The default value is $0.9$，$b_2$The default value is $0.999$ ,$ϵ$的$10^{-8}$，$Adam$ combines momentum and$RMSprop$ the merits of both, from experience has shown$Adam\; performs\; well\; in\; practiceand$ has advantages compared with other adaptive learning algorithms$.$

Summarize

Let's take a look at the performance of the above optimization algorithms on saddle points and contour lines:


As can be seen from Figure 6 and Figure 7, $Adagrad$、$Adadelta、RMS$ propRMSprop$\_$$\_$$\_$$\_$$RMSrop$ finds the right direction and moves forward almost quickly, and converges fairly quickly, while other methods are either slow or take many detours to find it$.$

1. First of all, there is no conclusion on which of the major algorithms is better. If you are just getting started, give priority to $SGD$ +$Nesterov$ $Momentum$ or$Adam$。

2、 A d a m Adam Adaptive learning rate algorithms such as$Adam$$have\; advantages\; for\; sparse\; data,\; and\; the\; convergence\; speed$$is$$SGDM$ tends to achieve better end results.

3. Choose according to your needs - in the process of model design experiment, to quickly verify the effect of the new model, you can first use $Adam$ performs rapid experimental optimization; fine-tunedSGD SGD$can\; be\; used\; before\; the\; model\; is\; launched\; or\; the\; results$$are\; released$$The\; SGD$ series optimization algorithm performs extreme optimization of the model.

4. Consider the combination of different algorithms. First use $Adaam$ for a quick descent before switching to$The\; SGD$ series optimization algorithm is fully tuned.