Gradient Descent (GD) algorithm

Gradient Descent (GD) algorithm

1. Gradient descent

(1) Basics

1. Gradient

$\triangledown f=(\frac{\partial f}{\partial x},\frac{\partial f}{\partial y},\frac{\partial f}{\partial z})$

2. Gradient descent

Gradient descent is a common method for minimizing hazard and loss functions. The direction of the gradient is actually the direction in which the function rises fastest at this point, and what is often required is the minimization of the loss function, so there is a negative sign.
In actual work, it is an ideal situation to obtain a set of loss functions that make the loss function reach the global minimum, and a more general situation is to make the accuracy of the model reach a certain minimum value within the acceptable range.

3. Gradient descent algorithm

$\theta_{1}=\theta_{0}-\alpha \triangledown J(\theta_{0});$
, where $\theta_{1}$ Sum $\theta_{1}$ Indicates the beginning and end of the position, $\alpha$ represents the step size or learning rate, $J(\theta)$ denotes $\theta$ A function of $θ$

The actual steps of the algorithm:

Initialize weights and biases with random values
Pass the input to the network and get the output value
Calculate the error between predicted and true values
For each neuron that produces an error, adjust the corresponding (weight) value to reduce the error
Repeat iterations until the optimal value of network weights is obtained

Code:

(2) Classification

1. Batch Gradient Descent (BGD)

$\theta_{t+1}=\theta_{t}-\alpha \triangledown J_{t}(\theta)$
假如以MSE作为损失函数，有
$\hat{y_{i}}=\sum_{j=0}^{m} w_{j} x_{i,j} (w_{0}=b, x_{i,0}=1)\\ J(w)= \frac {1} {2n} \sum_{i=1}^{n} ( \hat{y_{i}} -y_{i})^{2}\\ \triangledown J(w_{j})=\frac{\partial J(w)}{\partial w_{j}}=\frac{1}{n} \sum_{i=1}^{n} ( \hat{y_{i}} -y_{i})x_{i,j}\\ w_{j,t+1}=w_{j,t}-\alpha \triangledown J_{t}(w_{j}),t=1,2,...$
Among them, j is the number of features of the data, m is the total number of features, i is the number of samples of the data, n is the total number of samples, t is the number of iterations epoch, $\hat{y_{i}}$ is the predicted value, $y_{i}$ is the actual value, $J (w)$ isthe cost function

(1) Advantages: the direction of descent is the overall average gradient, and the global optimal solution can be obtained
(2) Disadvantages: the entire data sample set needs to be calculated, and the speed will be relatively slow
(3) Features:

For the update of parameters, all samples contribute, so the calculation is the maximum gradient, and the magnitude of an update is large
When there are not many samples, the convergence speed will be very fast

2. Mini-batch Gradient Descent (MBGD)

Each update uses k samples, and the distribution of the response samples to a certain extent
$\theta_{t +1}=\theta_{t}-\frac{\alpha}{k} \sum_{i=k(t-1)+1}^{kt} \triangledown J_{i,t}(\theta)$
假如以MSE作为损失函数，有
$\hat{y_{i}}=\sum_{j=0}^{m} w_{j} x_{i,j} (w_{0}=b, x_{i,0}=1)\\ J_{i}(w)= \frac {1} {2} ( \hat{y_{i}} -y_{i})^{2}\\ \triangledown J_{i}(w_{j})=\frac{\partial J_{i}(w)}{\partial w_{j}}=( \hat{y_{i}} -y_{i})x_{i,j}\\ w_{j,t+1}=w_{j,t}- \frac{\alpha}{k} \sum_{i=k(t-1)+1}^{kt}\triangledown J_{i,t}(w_{j}),t=1,2,3,...,[\frac{n}{k}]$
Among them, j is the number of features of the data, m is the total number of features, i is the number of samples of the data, n is the total number of samples, t is the number of iterations epoch, k is the number of batch samples batch size, yi ^ \hat{y_{i $\overset{y_{i}}{^}$ is the predicted value, $y_{i}$ is the actual value, $J_{i}(w)$ isthe loss function

(1) Advantages: It ensures the training speed and the accuracy of the final convergence
(2) Disadvantages: It is difficult to choose an appropriate learning rate

3. Stochastic Gradient Descent (SGD)

Each iteration updates the parameters, using only one random sample p
$\theta_{t+ 1}=\theta_{t}-\alpha \triangledown J_{i=p,t}(\theta),p \in [1,2,...,n]$
假如以MSE作为损失函数，有
$\hat{y_{i}}=\sum_{j=0}^{m} w_{j} x_{i,j} (w_{0}=b, x_{i,0}=1)\\ J_{i}(w)= \frac {1} {2} ( \hat{y_{i}} -y_{i})^{2}\\ \triangledown J_{i}(w_{j})=\frac{\partial J_{i}(w)}{\partial w_{j}}=( \hat{y_{i}} -y_{i})x_{i,j}\\ w_{j,t+1}=w_{j,t}- \alpha \triangledown J_{i=p,t}(w_{j}),其中p \in [1,2,...,n],t=1,2,...$
Among them, j is the number of features of the data, m is the total number of features, i is the number of samples of the data, n is the total number of samples, t is the number of iterations epoch, p is the order of random samples, yi ^ \hat{y_{i $\overset{y_{i}}{^}$ is the predicted value, $y_{i}$ is the actual value, $J_{i}(w)$ isthe loss function

(1) Advantages: only the gradient of one sample needs to be calculated, and the training speed is very fast
(2) Disadvantages: it is easy to jump from one local optimum to another, and the accuracy decreases
(3) Features:

When there are many samples, the convergence speed is fast
Each update uses one sample to approximate all samples, so the approximate gradient is calculated, even with interference, falling into a local optimal solution

3.1. Averaged Stochastic Gradient Descent (ASGD, SAG)

If MSE is used as the loss function, and only one random sample p is used to update the gradient, there is
$\hat{y_{i,t}}=\sum_{j=0}^{m} w_{j,t} x_{i,j} (w_{0}=b, x_{i,0}=1)\\ J_{i,t}(w)= \frac {1} {2} ( \hat{y_{i,t}} -y_{i})^{2}\\ \triangledown J_{i,t}(w_{j})=\frac{\partial J_{i,t}(w)}{\partial w_{j}}=( \hat{y_{i,t}} -y_{i})x_{i,j}\\ 初始化：\triangledown J_{i,t=1}(w_{j})=(\sum_{j=0}^{m} w_{j,t=1} x_{i,j}-y_{i})x_{i,j}\\ w_{j,t+1}=w_{j,t}- \frac{\alpha}{n} [\triangledown J_{i=p,t}(w_{j})+\sum_{i=1}^{n}\triangledown J_{i\neq p,t-1}(w_{j})]\\ 其中p \in [1,2,...,n],t=1,2,...$
Among them, j is the number of features of the data, m is the total number of features, i is the number of samples of the data, n is the total number of samples, t is the number of iterations epoch, k is the number of batch samples batch size, yi ^ \hat{y_ $\overset{y_{i}}{^}$ is the predicted value, $y_{i}$ is the actual value, $J_{i}(w)$ isthe loss function.

In the SGD method, although the problem of high computational cost is avoided, the effect of SGD is often unsatisfactory for large data training, because each round of gradient update is completely consistent with the previous round. Data has nothing to do with gradients. The stochastic average gradient algorithm overcomes this problem, maintains an old gradient for each sample in memory, randomly selects the pth sample to update the gradient of this sample, and keeps the gradients of other samples unchanged, and then obtains the gradient of all gradients average, which in turn updates the parameters.

3.2 Stochastic Gradient Descent with Momentum (SGD with Momentum) (SGDM)

3.3 SGDW

3.4 SGDWM

3.5 Cyclical LR

3.6 SGDR

2. Gradient descent optimization algorithm

(1) Supplementary theoretical knowledge

1. Exponentially weighted average

$V_{t}=\beta V_{t-1}+(1-\beta)\theta _{t}$

2. Bias correction in exponentially weighted average

$V_{t}=\frac{1}{1-\beta ^{t}}[\beta V_{t-1}+(1-\beta)\theta _{t}]\\ (V_{0}=0)$

3. Exponentially weighted moving average

It is mainly to correct the previous data, when $t\to \infty$ , $\beta ^{t}\to0$ , that is, it has little influence on the later data

4. Nesterov acceleration algorithm

5. Newton's method

(2) Momentum optimization algorithm

Introduce the momentum idea in physics, accelerate the gradient descent, the gradient descent is in the same dimension, the parameter update becomes faster, and when the gradient changes, the update parameter becomes slower, so that the convergence can be accelerated and the turbulence can be reduced

1. momentum momentum

When the parameters are updated, the previously updated direction
$v_{t+1}=\beta v_{t} is retained to a certain extent +\alpha \triangledown J(\theta_{t})\\ \theta_{t+1}=\theta_{t}-v_{t+1}$

(1) Features:

When the gradient direction changes, momentum can reduce the parameter update speed, thereby reducing the shock
When the gradient direction is the same, momentum can speed up parameter updates, thereby speeding up convergence

2. Nesterov Accelerated Gradient (NAG)

The improvement of momentum is to make a correction when the gradient is updated. The specific method is to add the momentum of the previous moment
$v_{t+1}=\beta v_{t}-\alpha \triangledown J(\theta_{t}+ \beta v_{t})\\ \theta_{t+1}=\theta_ {t}+v_{t+1}$
Disadvantages:
This algorithm will lead to extremely slow running speed, which is twice as slow as momentum, so in the actual implementation process, almost no one directly uses this algorithm, but uses a deformed version.

Derivation process:
Let $\theta'_{t}=\theta_{t}+ \beta v_{t}$ ，得
$\theta_{t+1}=\theta_{t}+\beta v_{t}-\alpha \triangledown J(\theta_{t}+ \beta v_{t})$
$\Rightarrow\theta'_{t+1}-\beta v_{t+1}=\theta'_{t}-\alpha \triangledown J(\theta'_{t})$
$\Rightarrow\theta'_{t+1}=\theta'_{t}+\beta[\beta v_{t}-\alpha \triangledown J(\theta_{t}+ \beta v_{t})]-\alpha \triangledown J(\theta'_{t})$
$\Rightarrow\theta'_{t+1}=\theta'_{t}+\beta^{2}v_{t}-\alpha(1+\beta) \triangledown J(\theta'_{t})$
$\Rightarrow\theta_{t+1}=\theta_{t}+\beta^{2}v_{t}-\alpha(1+\beta) \triangledown J(\theta_{t})$

变形版本：
$v_{t+1}=\beta^{2}v_{t}-\alpha(1+\beta) \triangledown J(\theta_{t})\\ \theta_{t+1}=\theta_{t}+v_{t+1}$

insert image description here

Please add a picture description
Momentum: (blue) first update the gradient, and then take a big step in the original momentum direction
NAG: (green) first take a big step in the original momentum direction (brown), then calculate the gradient (red), and get the corrected green line

(3) Propagation optimization algorithm

1. Resilient propagation (Rprop)

2. Root Mean Square propagation Root Mean Square propagation (RMSprop)

A special case of Adadelta, when ρ=0.5, E becomes the average of the sum of squared gradients; if you find the root again, it becomes RMS (root mean square) gt = ▽ J ( θ t ) G t =
$g_{t}=\triangledown J(\theta_{t})\\ G_{t}=\beta G_{t-1}+(1-\beta)g_{t}^{2}\\ \theta_{t+1}=\theta_{t}-\frac{\alpha}{\sqrt{G_{t }+\varepsilon}} g_{t}$

(1) Features: RMSprop is a development of AdaGrad, a variant of Adadelta, the effect tends to be between the two. It is suitable for dealing with non-stationary targets and works well for RNN

(4) Adaptive learning rate algorithm

1. Adaptive Gradient (AdaGrad)

$g_{t}=\triangledown J(\theta_{t})\\ G_{t}=G_{t-1}+g_{t}^{2}\\ \theta_{t+1}=\theta_{t}-\frac{\alpha}{\sqrt{G_{t}+\varepsilon}} g_{t}$
where $\alpha$ is the learning rate, generally $0.01$ ； $\varepsilon$ is to prevent the denominator from being 0, generally take $10^{-7}$

(1) Advantages: In the early stage, in the direction of a more gentle parameter space, greater progress will be made; for parameters with larger gradients, the learning rate will become smaller. For parameters with small gradients, the effect is reversed. In this way, the parameters can be reduced slightly faster in a gentle place, so as not to stagnate.
(2) Disadvantages: the learning rate is reduced prematurely and excessively; because it is the square of the accumulated gradient, the gradient will disappear in the later stage

2. Adadelta

AdaGrad constraints
$Theorem:E(g^{2})_{t=0}=0,E(h^{2})_{t=0}=0 \\g_{t}=\triangledown J(\theta_{t})\\E(g^{2})_{t}=\beta E(g^{2})_{t-1}+( 1-\beta) g_{t}^{2}\\h_{t}= \frac {\sqrt{E(h^{2})_{t-1}+\varepsilon}}{\sqrt{E (g^{2})_{t}+\varepsilon}} g_{t}\\ E(h^{2})_{t}=\beta E(h^{2})_{t-1 }+(1-\beta) h_{t}^{2}\\\theta_{t+1}=\theta_{t}-h_{t}$
where $\varepsilon$ is to prevent the denominator from being 0, generally take $10^{-6}$

(1) Features:

In the early and mid-term of training, the acceleration effect is good, very fast
late training. Repeatedly jitter around local minima
Do not rely on the global learning rate, manually setting a global learning rate will not affect the final result

3. Adaptive Momentum Estimation (Adam)

$初始化：m_{0}=0,v_{0}=0\\ g_{t}=\triangledown J(\theta_{t})\\ m_{t}=\frac{1}{1-\beta_{1} ^{t}}[\beta_{1} m_{t-1}+(1-\beta_{1})g_{t}]\\ v_{t}=\frac{1}{1-\beta_{2} ^{t}}[\beta_{2} v_{t-1}+(1-\beta_{2})g_{t}^{2}]\\ \theta_{t+1}=\theta_{t}-\frac{\alpha}{\sqrt{v_{t}}+\varepsilon} m_{t}$
where $\alpha$ is the learning rate, generally $0.001$ ; $\beta_{1}$ and $\beta_{2}$ respectively $0.9$ and $0.999$ $_$ $0.999$ , $\varepsilon$ is to prevent the denominator from being 0, generally take $10^{-8}$

(1) Explanation:

Adam also needs to find the sum of gradient squares v (that is, G in RMSPropz)
Smoothly update the gradient using the new variable m
The sum of squared gradients of the gradient will be updated with reservations
The sum of squared gradients of the gradient will be updated with reservations

(2) Features:

The parameters are relatively stable
Good at dealing with sparse gradients, good at dealing with non-stationary targets
Compute different adaptive learning rates for different parameters
Also suitable for most non-convex optimization problems - suitable for large data sets and high-dimensional spaces

3.1 Adamax

Adam added learning rate cap

3.2 AdamW

Adam joins weight decay

3.3 AMSGrad

3.4 I hope

Introduce Nesterov acceleration effect in Adam

3.5 SparseAdam

Adam for sparse tensors

3.6 AdaBound

(5) Others

1.SWATS

2. RAdam

3.Lookahead

4.Nesterov accelerated gradient (NAG)

method comparison

algorithm	advantage	shortcoming	Applicable situation
batch gradient descent	When the objective function is convex, the global optimum can be found	The convergence speed is slow, all the data needs to be used, and the memory consumption is large	Not suitable for large data sets, cannot update the model online
stochastic gradient descent	Avoid the interference of redundant data, accelerate the convergence speed, and be able to learn online	The variance of the update value is large, and the convergence process will fluctuate, which may fall into a minimum value. It is difficult to choose an appropriate learning rate	Suitable for models that need to be updated online, suitable for large-scale training samples
Mini-batch gradient descent	Reduce the variance of the update value, and the convergence is more stable	Choosing the right learning rate is difficult
Momentum	Ability to speed up SGD in relevant directions, suppress oscillations, and thus speed up convergence	Need to manually set the learning rate	Suitable for reliable initialization parameters
Nesterov	The gradient is calculated to correct the current gradient after a large jump	Need to manually set the learning rate
Dosing	No need to manually adjust each learning rate	It still relies on manually setting a global learning rate. If the learning rate is set too large, the adjustment to the gradient is too large. In the middle and late stages, the gradient is close to 0, which makes the training end early	Fast convergence is required, when training complex networks: suitable for processing sparse gradients
Adadelta	There is no need to preset a default learning rate. In the early and middle stages of training, the acceleration effect is good and fast, which can avoid the problem of inconsistent units on both sides when updating parameters	In the later stage of training, it repeatedly jitters near the local minimum	Fast convergence is required, when training complex networks
RMS plug	Fix aggressive learning rate scaling in Adagrad	Still depends on the global learning rate	Fast convergence is required, when training complex networks; good for non-stationary targets - works well for RNNs
Adam	Smaller memory requirements, calculate different adaptive learning rates for different parameters		Fast convergence is required when training complex networks; the advantages of being good at handling sparse gradients and dealing with non-stationary targets are also suitable for most non-convex optimizations - suitable for large data sets and high-dimensional spaces

For sparse data, try to use an optimization method with adaptive learning rate
SGD usually takes longer to train, but results are more reliable
If you care about faster convergence, it is recommended to use the learning rate adaptive optimization method
Adadelta, RMSprop, Adam are relatively similar algorithms

The above is not finished and needs to be updated, it is only for personal study, the infringement contact is deleted, if there are any mistakes or deficiencies, please point out for improvement.

Deep learning--gradient descent algorithm (continuous update)

Table of contents