Series notes | deep learning serial (5): optimization techniques (under)

Click on the top " AI proper way ", select the "star" Public No.

Heavy dry goods, the first time served

Depth study we conclude that five tips:

This section continues to talk from the start of the third.

3. Early stopping and Regularization

In this section we discuss with Early stopping and Regularization, these two skills are not unique to the depth of learning methods, machine learning is the common method.

Early stopping

During training, often with the conclusion of training also may not be as before, the reason is likely to occur overfitting. We need to advance on the brakes, draw better results.

Regularizaton

When we tried to reduce the value Loss of function, we will find that we find a set of parameters weights, not only to make Loss and smaller, and their weights also need to be close to zero, so that our results will be even better.

L1 regularization:

New Loss function will be minimized:

L2 Regularization:

New Loss function will be minimized:

Here, many students may wonder why the small weights, the result it is better, I am here to illustrate: 6 years old and 14 years old, the brain's neuronal density was significantly decreased, indicating that some of the neurons are invalid hinder the progress of the brain.

4. Dropout

Dropout imagenet game shine in 2012, is one of CNN model was part of feats to win.

So what is our first intuitive understanding Dropout:

When practicing martial arts, weight training when feet tied

Wait until excel down time:

We have several ways to explain Dropout

Basic definitions

When training, each neuron may have p% "step aside"

When tested, all the neurons come together to:

Dropout is a learning Ensemble

Ensemble learning we discuss in machine learning through column, the link is an integrated learning. Each training when the network structure is not the same, is a thinner network: