When training a neural network, we have a training set and a test set, and people are almost infallible about this
When the training set accuracy is 99% and the test set accuracy is 90%
At this time, there is overfitting, and we call it high variance.
When the training set accuracy is 85% and the test set accuracy is also 85%
This is the bad fit, we call it high bias
For the convenience of expression later, I call the training set precision a and the test set precision b
High variance solution:
When a high variance occurs, a and b first rise together, and then after a certain point, a continues to rise and b begins to fall.
What I usually use is to output the training set accuracy every 100 times of training, and save the model every 1k times
Once b starts to drop, stop training
What is defined here as b starts to fall
It doesn't mean that b starts to decline after 300 consecutive training sessions.
It is best to train 1000 - 2000 times in a row that b is falling before it can be called b has started to fall
because