Deep learning-debugging notes for deep neural networks

1 Quick verification of the model

1.1 Make sure the model can converge

At the beginning of debugging, it may be difficult to determine whether the model has converged. At this time, when debugging, you can appropriately increase the order of magnitude of the learning rate to see if the loss of the model increases with the number of iterations during the first few iterations. A drop of about 10 times. If you observe that the value of loss can drop at a visible rate of "about 10 times", it means that the model can converge. At this time, the design of the loss function is likely to be normal, and the learning rate can be adjusted back to normal Value continues to be tested;

2 Use of Early Stop Strategy

2.1 Evaluation indicators of Early Stop

The cumulative value of the loss value;

MAP of the training set;

MAP of the test set;

2.2 Early stop exploration times-26 times

We set the number of explorations for Early stop to 26. How did we get this?

We first assume that the probability of successful and unsuccessful exploration is 0.5, and we set the threshold of exploration action to 99.999%,

The method for calculating the number of explorations is as follows:

Therefore, we chose a number that is easy to remember, which is 26 times;