1. Model Assessment (Model Assessment)
The notes are from "Mathematics of Vernacular Machine Learning"
After we have trained the model, we need to know whether the model is good or not. Quantitatively describing the quality of the model is model evaluation.
Divide all the acquired training data into two parts: one for testing and one for training. The former is then used to evaluate the model. In most cases, a ratio of 3:7 or 2:8 is used for training data.
The practice of dividing all training data into test data and training data is calledCross-validation
If we want to do 4-fold cross-validation, then we will measure the accuracy like this. It
will be very time-consuming to increase the value of K unrealistically, so we must determine an appropriate value of K
1.1 Verification of the regression problem
For the case of regression, just calculate the square of the error of the test data on the trained model, and then take the average. (In fact, it is an error function, and the objective function of the regression problem is also an error function)
There are n test data, then the mean square error (MSE)
The smaller the MSE, the higher the accuracy and the better the model
In addition to MSE, there are root mean square error (RMSE), mean absolute error (MAE), root mean square logarithmic error (RMSLE), etc.
1.2 Validation for classification problems
Accuracy
Indicates the proportion of correctly classified data TP and TN in the entire data set, and the obtained accuracy can basically grasp the overall accuracy of the classification results
Precision(Based on TP)
Among the data classified as Positive, the proportion of True data
Precision(mainly TN)
Recall rate (Recall)(Based on TP)
Among the True data, the proportion of the data that is actually classified as Positive
Recall rate (Recall)(Typically based on TN)
How to choose whether to use TP-based precision and recall or use TN-based?
When the data is unbalanced, it is better to use the one with a smaller number.
If there are few Positives, use TP-based ones
. If there are few Negatives, use TN-based ones.
Usually, the precision rate and the recall rate will be high and low, and we need to make a trade-off, which is a bit troublesome
F1 value
Harmonized mean of precision and recall
Weighted F-value
β \betaβ is the weight, and when the weight is 1, it is the above formula
