Neural network model optimization-training optimization

Batch training

Cut the training data into many batches, and then feed the model batch by batch;

advantage

1. Increase training speed;
2. Introduce to the training processRandomness；

This oneRandomnessThe most important thing is that it can effectively solve the local optimum. I now divide it into many batches. The initial position of each batch is very random. It is like sprinkling beans. Sprinkle a little in each place, so it is not easy to get stuck in a certain small area. pit.

The red part represents the batch data, so that almost every local optimum (green) can be stepped on, and then the global optimum can be found.

Gradient descent with momentum optimizer

Too much randomness has a disadvantage. Not every iteration will go in a good direction, it will cause the calculation time to be too long. At this time, the parameter β can be introduced to introduce inertia.

The larger β is, the smaller (1-β) is, and dW has an effect on $v_w$The smaller the impact, which is equivalent to $v_w$The greater the inertia, the harder it is to change, $v_b$ is also like this.

RMSProp optimizer

The first formula is dW ² , and the previous optimizer $dW$ is different.
When W is updated, the amount of change is divided by the root sign$S_w$

Compared with the previous pure inertia, which stumbles towards the global optimal solution, this optimizer will move to the global optimal solution more steadily.

advantage

Relatively stable

Adaptive moment estimation Adam optimizer

It can be simply understood as the combination of the previous two optimizers, introducing the parameters
$$b_1,b_2,\epsilon ,t$$
$b_1$Often take 0.9, $b_2$0.999 is often taken,
where $\epsilon$ is a very small number, often 1e-8, 1e-9, to prevent the explosion.
$t$ is the number of iterations, which is equivalent to considering the pace, which can be known from the following formula, so that$v_w$, $v_b$When changing, the steps are large at the initial moment, and the global optimal solution is quickly moved forward, and at the end, it is a little more cautious and slowly moves towards the global optimal solution.

In addition, the parameters are not necessarily these values, and need to be carefully adjusted by the big guys.

advantage

It is suitable for large-scale data, and can also solve problems such as high noise and sparse gradient.