Starting from this blog post, there will be several blog posts exploring how to evaluate the model, is the model good or bad? Does the model fit?
- P20 (this article) model diagnosis, validation set
- P21 Regularization
- P22 Exploration of Overfitting and Underfitting 2, Bias and Variance
Model Evaluation - Validation Set
The streamlined version of this blog post:
the verification set is used to judge whether it is overfitting or underfitting. Usually, the data set is allocated as: 60% training set + 20% verification set + 20% test set. During model.fit training, the verification set is Like an eye, it will always observe whether the training of the model will lead to overfitting and underfitting.
Why do you need an evaluation model
In machine learning, we use models to make predictions and classify data. The quality of the model usually refers to the prediction accuracy and generalization ability of the model.
The predictive accuracy of a model is the ability of the model to produce the correct output given the input data. Usually different indicators are used to measure the prediction accuracy of the model, such as precision, recall, F1 score, etc.
The generalization ability of the model refers to the ability of the model to perform on unseen data. The model should be able to correctly predict the output for unknown data. A model is said to be overfitting if it performs well only on the training dataset but poorly on unseen data. An overfit model will pay too much attention to the details of the training data, resulting in the inability to make accurate predictions on new data. Conversely, if the model performs poorly on both the training and test datasets, then we say the model is underfitting. An underfit model fails to make predictions correctly on both the training and test datasets.
Whether it is in the regression model or in the classification model task, there will be overfitting and underfitting phenomena:
Therefore, a good model should perform well on both the training dataset and the test dataset, with high prediction accuracy and good generalization ability. The validation set is a method commonly used in machine learning to evaluate model performance.
What is a validation set
Validation set, validation set, also known as cross validation set, cross validation set, is a data set used to evaluate the performance of machine learning models.
When training a model, the data set is usually divided into three parts: training set, validation set and test set. The training set is used to train the model, the test set is used to evaluate the performance of the final model, and the verification set is used to adjust the hyperparameters of the model and select the best model, as shown in the figure below:
❓The first doubt often arises here: "There is a test set, why do we need a verification set?"
Validation and Test Sets
Stream saving: Both the verification set and the test set are used to evaluate the model, but the verification set is during model training, and the test set is after model training;
The simplest model training process:
① Select model
② Adjust parameters for training
③ Gradient descent reduces loss value
④ Repeat ②~③
⑤ Output model, test set judgment accuracy
Non-flow-saving:
The role of the validation set is to help tune the hyperparameters of the model and select the best model. During the training process, the verification set data does not participate in the update of model parameters, but is used to evaluate the generalization ability of the current model. By observing the performance indicators on the validation set, we can judge whether the model has overfitting or underfitting problems, and adjust the model accordingly to improve the performance of the model.
The function of the test set is to evaluate the performance of the final model after the model training and adjustment are completed, and to simulate the performance of the model in practical applications. The test set should be a data set independent of the training set and validation set, and the test set data cannot be used to make any adjustments during the training process, otherwise the test set will lose its independence and the performance of the model cannot be truly evaluated.
Although the data in the test set can be used to judge whether the model is overfitting, the data in the test set can only be used after training. Moreover, the data of the test set cannot be used to adjust the model, otherwise it may lead to deviation of the test set and affect the accuracy of the final evaluation result. Therefore, in the training process, the role of the verification set is very important, it can help adjust the hyperparameters of the model and select the best model, so as to avoid problems such as overfitting.
❓At this time, the second question comes, how does the verification set participate in the neural network? This raises two questions:
- What is the role of a validation set in a neural network?
- How to tune neural network parameters using validation set?
Validation set and neural network
What is the role of the validation set in a neural network
Although the validation set cannot automatically adjust the model or directly affect the training process of the model, it still plays an important role in model selection, parameter tuning and evaluation.
- Model Selection
First, the role of the validation set in model selection is to help select the best model. After training multiple models, the validation set is used to evaluate the performance of each model, and the best performing model is selected as the final model. - Parameter tuning of the model
Second, the role of the validation set in terms of hyperparameter tuning is to help choose the best combination of hyperparameters. When training multiple versions of the same model, use the validation set to evaluate the performance of each version and choose the version with the best combination of hyperparameters. - Model Evaluation
Finally, the role of the validation set in model evaluation is to provide an independent assessment of model performance. Before using the test set to evaluate the performance of the model, it is usually necessary to use the validation set to adjust and optimize the model to ensure that the evaluation results of the test set are accurate.
Therefore, although the validation set cannot directly affect the training process of the model, it still plays an important role in model selection, parameter tuning, and evaluation.
❓The selection of the model is mentioned above, so what is the selection of the model, and how to choose the model?
Neural Network Model Selection
Choose model 1, or model 2, or model 3?
In neural networks, model selection usually refers to choosing different neural network structures, such as different numbers of layers, nodes, activation functions, etc., to meet different task requirements and data characteristics. Therefore, in neural networks, the choice of model is very important.
Although many classic neural network structures have appeared in the field of deep learning, such as LeNet, AlexNet, VGG, ResNet, Inception, etc., in specific applications, it is still necessary to select a suitable neural network structure according to specific tasks and data characteristics. Different neural network structures may have different advantages and disadvantages, for example, some structures may have better classification performance, while others may be more suitable for processing sequence data or image data, etc.
When choosing a model, the following factors generally need to be considered:
-
Task requirements: According to the type and requirements of the task, select the appropriate neural network structure. For example, for image classification tasks, convolutional neural network (CNN) structures are usually used, while for sequence data processing tasks, recurrent neural network (RNN) structures are usually used.
-
Data characteristics: According to the characteristics and scale of the data, select the appropriate neural network structure. For example, for image data, if the data size is small, you can choose a shallow CNN structure; if the data size is large, you can choose a deep ResNet structure.
-
Computing resources: According to the limitation and availability of computing resources, choose a suitable neural network structure. For example, in the case of relatively limited computing resources, you can choose a lightweight neural network structure, such as MobileNet, ShuffleNet, etc.
In short, in neural networks, the selection of models is still very important, and needs to be selected according to factors such as specific task requirements, data characteristics, and computing resources.
How to tune neural network parameters using validation set
How to tune hyperparameters based on validation set? You can observe the loss function curve of the validation set:
In the figure J cv J_{cv}JcvFor the validation set loss function curve J train J_{train}JtrainFor the training set loss function curve
The loss function of the verification set will have a fluctuation curve from large to small, and then large again. The first one from large to small is because the fitting effect of the entire model is continuously improved with the gradient descent and parameter adjustment; and The second one is from small to large because with continuous adjustment, there will be over-fitting phenomenon, resulting in an increase in the loss value of the verification set.
In fact, for the content of this chapter, you only need to know that there is a set called the verification set, which is used to participate in the parameter adjustment process during the model fitting process, and the purpose is to observe whether there is overfitting or underfitting. This article lacks specific examples of overfitting and underfitting and how to solve them after they appear. This will be in the blog post:
- P22 Exploration of Overfitting and Underfitting 2, Bias and Variance
explained in.