1. Summary
One sentence summary: the role of the verification set is to adjust the hyperparameters
1. Hyperparameters?
[The value of the hyperparameter is not learned]: Most machine learning algorithms have hyperparameters, which can be set to control the behavior of the algorithm. The values of the hyperparameters are not learned by the learning algorithm itself.
[Hyperparameters if learning is too difficult to optimize]: Sometimes an option is set as a hyperparameter that the learning algorithm does not need to learn because it is too difficult to optimize. More often than not, this option must be a hyperparameter because it is not suitable for learning on the training set.
[Hyperparameters are set by ourselves]: This applies to all hyperparameters that control the capacity of the model. If hyperparameters are learned on the training set, these hyperparameters always tend towards the maximum possible model capacity, leading to overfitting.
2. How to solve the problem of hyperparameter setting?
[Validation set sample]: In order to solve the problem of hyperparameter setting, we need a validation set sample that cannot be observed by the training algorithm.
3. Does the test set not help improve hyperparameters?
[The test sample cannot participate in the selection of the model in any form, including the setting of hyperparameters]: The test set can be used to estimate the generalization error of the learner after the learning process is completed. The key point is that the test sample cannot be used in any form. Participate in the selection of the model, including the setting of hyperparameters. For this reason, samples from the test set cannot be used for the validation set.
[Validation set helps improve hyperparameters]: Therefore, we always build a validation set from the training data. It is used to estimate the generalization error during or after training and update hyperparameters.
4. What are the general hyperparameters of the model?
such as learning rate and features etc.
5. Is the validation set validation or training?
The training set is used to train the parameters of the model, and the validation set is used to train the hyperparameters of the model. Different hyper-parameter combinations correspond to different latent models. What runs on the verification set is actually a collection of models. The existence of the verification set is to find the best performing model from this bunch of possible models.
6. The importance of hyperparameters?
Hyperparameters include number of training epochs, learning rate, etc. If a model has much more training data than others, the effect will be very good (the parameters are well trained), but the model architecture (hyperparameter settings) may not be better than others.
7. Why can't I train hyperparameters on the training set?
For example, the number of training rounds (epochs), on the same training set, training 3 rounds and training 10 rounds, the result is definitely a different model. Their parameters are not the same. So is it better to train for 3 rounds or 10 rounds? Or both are bad and should be trained for 6 rounds? This kind of decision can only be seen on the verification set after training. Generally, after training for several epochs, run a verification to see the effect. If you find that the effect of training for 3 rounds is better, then you should discard the potential model for training 6 rounds and 10 rounds, and only use the results of 3 rounds of training.
Another example is the number of network layers. When training on the training set, the more layers, the better the fitting effect. If you select your model in this way, the effect on the verification set must be much worse, which is overfitting. Therefore, it is necessary to take a part of the samples from the training samples, that is, the verification set, so that these samples cannot be seen during the model training process, and then evaluate the performance of the model in this part of the samples, and finally select the appropriate number of network layers.
You can also draw inferences about other hyperparameter selections. In short, the hyperparameters are selected according to the effect of the verification set to determine the final model.
To sum up: it is equivalent to manual parameter adjustment on the verification set (on the training set, it is automatically adjusted through error backpropagation)
In the next step, the model is submitted to the test set for verification. The test set should be different from both the training set and the validation set. As for the final selection model, no one can guarantee how well it performs on the test set. Just like you did well in the mock exam, the college entrance examination may also fail.
How to divide the validation set
How to set the verification set and how much data to divide for verification are actually decisions that each researcher needs to make independently, and should not be forced to be consistent.
10-fold cross validation (10-fold Cross Validation)
Selection of fixed training and test sets may cause partition bias
Therefore, the data set is randomly divided into 10 parts, 9 of which are used for training and the other 1 is used for testing. The process can be repeated 10 times, each time using different test data.
The advantage is that the error is reduced by random division, and the second is that 90% of the training data is used each time instead of only 50% of the data in the 2-fold cross-validation.
2. Hyperparameters and validation set
Most machine learning algorithms have hyperparameters that can be set to control the algorithm behavior. The values of the hyperparameters are not learned by the learning algorithm itself.
Sometimes an option is set as a hyperparameter that the learning algorithm does not learn because it is too difficult to optimize. More often than not, this option must be a hyperparameter because it is not suitable for learning on the training set. This applies to all hyperparameters that control model capacity. If hyperparameters are learned on the training set, these hyperparameters always tend towards the maximum possible model capacity, leading to overfitting.
To solve this problem, we need a validation set sample that was not observed by the training algorithm.
The test set can be used to estimate the generalization error of the learner after the learning process is completed. The key point is that the test samples cannot participate in the selection of the model in any form, including the setting of hyperparameters. For this reason, the samples in the test set Cannot be used for validation set. Therefore, we always construct the validation set from the training data. In particular, we split the training data into two disjoint subsets. One of them is used to learn the parameters. The other is used as a validation set to estimate the generalization error during or after training and to update hyperparameters. The subset of data used to learn parameters is often still called the training set, although this will be confused with the data set used for the entire training process. The subset of data used to pick hyperparameters is called the validation set.
The process of using test and training sets to drive model development iterations. At each iteration, we train on the training verses and evaluate the test data, and use the evaluation results based on the test data as a guide to choose and change various model hyperparameters, such as learning rate and features. Is there a problem with this approach? ?
Existence, the more we perform evaluations on a given test set, the higher the risk of unknowingly overfitting to that test set.
Dividing the data set into three parts can greatly reduce the chance of overfitting.
This workflow is better because it exposes less information to the test set.
Constantly using test and validation sets will gradually make it lose its effectiveness. That is, the more you use the same data to decide on hyperparameter settings or other model improvements, the less confident you can be that those results will actually generalize to new, unseen data. Note that validation sets generally fail more slowly than test sets.
If possible, it is recommended that you collect more data to " refresh " the test and validation sets. Starting over is a great way to reset.