1. Set aside method: Divide the full data set into two disjoint parts, of which the larger part (generally 2/3 to 4/5 of the total data volume) is used as the training set, and the other part is used as the test set . When dividing the data, the consistency of the data distribution in the training set and the test set should be maintained (methods such as stratified sampling can be used); at the same time, considering the influence of the randomness of the division, the division should be repeated many times.
2. Cross-validation method (k-fold as an example): Divide the full data set into k parts that are disjoint and have the same amount of data, and perform k model evaluations. The i-th time (i=1,2,...,k) takes the i-th data as the test set, and the rest of the data as the training set. The results of k model evaluations are averaged as the final model evaluation result. The k-fold cross-validation when k is equal to the amount of data is called leave-one-out method.
3. Bootstrap self-help method: From the original full data set (sample size is m), a new sample with a sample size of m is extracted by the method of repeated sampling with replacement. When m is large, the probability that a sample in the original data set is not selected in m times of extraction (that is, the probability that a certain data does not enter the new sample) is about 0.368, so there are about 36.8% of the data in the original data set. Not in the new sample. Take the new sample as the training set, and use the remaining 36.8% of the data as the test set.
Comparison of the three:
1. The bootstrap method is more advantageous when the amount of original data is small, but since the bootstrap method changes the distribution of the data, it may lead to estimation bias. Therefore, when the amount of data is large, the leave-out method and cross-validation method are generally preferred.
2. With the increase of the number of repeated divisions in the leave-out method/the complexity of stratified sampling, the increase of k in the cross-validation method/even the leave-one-out method, the workload of model evaluation may be very large, and the high precision is the amount of calculation. for the price.
3. In practice, an appropriate model evaluation method should be selected based on comprehensive consideration of data volume, equipment, budget, etc.