Model training method for small data volume

In many cases, due to the limitation of current data or the sensitivity of the data, the data obtained is very small. For the case of too little training data, the following methods can be considered

1. Transfer learning
   By using pre-trained models that were previously trained on large data sets, we can directly use the corresponding structure and weights to apply them to the current problem. The pre-trained model may not use the same type of data . This is called "transfer learning", which is to "transfer" the pre-trained model to our problem. For example, we have learned to play basketball, and now want to learn to play volleyball, then we will learn the skills and experience gained when playing basketball. Move to playing volleyball.

2. Feature extraction + feature fusion + xgboost
   When there are few data sets, you can consider using a pre-trained model for feature extraction (without any training), and use the extracted features as input to xgboost (xgboost is a lifting tree model, which will Many tree models are integrated together to form a strong classifier, which has a strong advantage in dealing with unbalanced data or preventing overfitting, and processing high-dimensional features. Other classifiers can also be used here). The effect will be better than using migration directly. Study better.
   After feature extraction is used, a feature fusion strategy can be adopted. A simple understanding is to stitch various features together and put them into XGBoost

Why this method is better than direct transfer learning
Reasons:
1) A small number of data sets makes it easy to overfit the model in transfer learning.
In transfer learning, due to the pre-model made by the large amount of data directly used in the past, it may cover many other Features, so when using transfer learning directly, some features that are not obvious in the data are also amplified in transfer learning, so that such features are also extracted, resulting in over-fitting of the model.
2) A small number of data sets make the model in a certain A local minimum value, so that the ability of the model to extract features is not as good as the ability of the PreTrained model that has been heavily debugged.

3. Data Enhancement
   For data enhancement methods, you can generally invert, rotate, increase contrast, cut and enlarge, etc., so that a training picture can be derived from multiple pictures, thereby increasing the sample data. However, there are also disadvantages, that is, after the data is enhanced, it is likely to introduce noise, or completely wrong data

4. Transcendental knowledge
   Transcendental knowledge is also a type of transfer learning, which transfers the weights learned by other models to its own model, and transcendental knowledge is to add our human knowledge to modeling, more like Is a way to guide model learning

References:
https://zhuanlan.zhihu.com/p/50547038