First, the process generally enhance the performance of the model in four directions:
1.Data Augmentation
2.Weight Initialization
3.Transfer learning + Fine-tune
4.Ensemble / Model Fusion
Data enhancement, transfer learning and fine-tuning these previously taught you. Heavy constraints on the right to regularization. Because of the growing amount of data alone, it is not enough. Because the upgrade over the data are highly correlated. Against over-fitting model should focus on the "entropy capacity" is the model allows the amount of information stored. 1. The storage capacity, can use more features, better performance. 2. Storage small amount, then we want to store features focused on truly relevant features, and better generalization performance.
At this time, correct adjustment need "entropy capacity":. A number of parameters to adjust the model selected, number of layers each comprising a scale model. B. weights are regularization constraints, such as L1 or L2 will constraint model heavy weight shifted to a smaller value. C. + dropout lift data are associated in the random scrambling of the data. During the migration study, the weight of the heavy load initialization is good right parameters have been trained to initialize the weights as at this time.
Fusion model, it is advantageous that substantially no disadvantages, can substantially increase the performance of 1% -2%.
a. fusion results from the documents submitted, if several models are quite different, then, the final model will get a larger raise.
B. Fusion weighted voting. That will be more emphasis on good performance models, need to give greater weight.
. C averaged the results: averaging approach can often reduce the over-fitting phenomenon.
Second, the training set, classification and verification role set, test set
My personal understanding is:
Training set: used to train model, mainly to update the weight parameters to the model after each batch of the training set, has been further optimized, better performance.
Validation set: validation set is generally concentrated from the extracted training data representative of the model to verify the quality of training. Its main role, authenticated demerit, updated over the parameters, such as the use of lower learning rate, increase the number of iterations, etc., can also be observed whether over-fitting phenomenon.
Test set: is the final test of the model used for good or bad.
In short: the training set, validation set, the test set are independent of each other, there is no intersection. Validation set is generally concentrated from the beginning of the training division out.
Third, the concept of batch_size
Batch selection, the direction of decline is first determined if the data set is small, entirely in the form of the full data set may be employed. There are at least two advantages: a direction indicated by the direction of the full data set can be determined more representative sample population, thereby accurately located toward the extreme value. 2. Since the gradient value difference larger weight different weights, thus selecting a global learning rate is difficult.
Within a certain range, Batch_size general, the more quasi which determines the direction of decline, the smaller the shock caused by the training. In this case, the method can be increased to improve the accuracy of epochs.
----------------
Original link: https://blog.csdn.net/weixin_37203756/article/details/80185742