Predictive performance evaluation index of basic knowledge of machine learning

Predictive Performance Evaluation Ideas

Usually, the difference between the predicted output value on the training data set and the corresponding real value in the training data set is called "training error", and the predicted value obtained by using the trained model to test on the test data set and The difference between the real values ​​is called "generalization error". The biggest purpose of using the prediction model for big data prediction is to keep the "generalization error" at a minimum value, but in the actual process, only through the training of the model The process makes the "training error" as small as possible, so in practical applications, it is almost impossible for the "generalization error" between the predicted value and the real value to be 0 when testing on the test data set.
Due to the unavoidable error, when evaluating the quality of the model, it is necessary to use a series of indicators to evaluate its performance. When evaluating the performance of the prediction model, it is mainly to evaluate the predicted value of the trained model on the new data set. To evaluate whether it is good or bad, some mathematical and statistical formulas are usually used to calculate the relationship between the predicted value of the model and the actual real value. Common performance evaluation indicators include mean square error, root mean square error, mean absolute error, mean absolute percentage error, symmetric mean absolute percentage error, and degree of fit. The following will introduce them respectively.

1. Mean Square Error (Mean Square Error, MSE)

The value range of this index is from 0 to positive infinity. When the predicted value of the model is exactly the same as the real value, the value is 0. Otherwise, the larger the difference between the predicted value and the real value, the larger the value. Its calculation formula is as follows:

Where n is the number of samples, y_p is the predicted value, and y_t is the real value.

2. Root Mean Square Error (RMSE)

This indicator mainly takes the square root of the mean square error, which makes it more intuitive to observe the error on the order of magnitude. The value range of this indicator is the same as the mean square error, from 0 to positive infinity. When the predicted value of the model is different from the real value When they are exactly the same, the value is 0; otherwise, the greater the difference between the predicted value and the real value, the greater the value. Its calculation formula is as follows:

3. Mean Absolute Error (Mean Absolute Error, MAE)

The difference from the previous two indicators is that this indicator does not need to calculate the square of the error value. This indicator mainly calculates the average value of the absolute value of the error between the predicted value of all forecast samples and the real value to judge the quality of the forecast. Its calculation formula is as follows:

The same as the previous two indicators, the value range of this indicator is also from 0 to positive infinity. When the predicted value of the model is exactly the same as the real value, the value is 0, otherwise the difference between the predicted value and the real value The larger the difference, the larger the value.

4. Mean Absolute Percentage Error (MAPE)

The previous indicators use real numbers as the standard for evaluating the effect of forecast error, while the average absolute percentage error uses percentage as the standard for evaluating the effect of forecast error. The value range of this indicator is from 0 to positive infinity. When the value of this indicator is When it is 0%, it means that the predicted value under the model is completely consistent with the real value, and the model can be regarded as a perfect model. poor predictive effect. The formula for calculating this indicator is as follows:

It can be seen that the formula uses the real value as the denominator part, which is similar to normalizing the error value between the predicted value and the real value, so as to avoid the influence of some error extreme points on the absolute error, but due to In the presence of this feature, when there is 0 in the real value of the data set, the predictive evaluation index cannot be adopted.

5. Symmetric Mean Absolute Percentage Error (SMAPE)

Same as the average absolute percentage error, this indicator also uses percentage as the standard for evaluating the effect of forecast error. The effect of the value point on the absolute error. Similarly, when there is 0 in the real value of the data set, the predictive evaluation index cannot be adopted. The formula for calculating this indicator is as follows:

It can be seen from the above formula that the value range of this indicator is from 0 to positive infinity. The smaller the calculation result is, the smaller the difference between the predicted value of the model and the real value is, and the better the prediction effect of the model is.

6. Degree of fit (R-squared)

The degree of fit is mainly used to measure the degree of fit between the predicted value of the model and the real value, and its calculation formula is as follows:

In the above formula, y ̅_t is the average of the true values.
The biggest feature of the fit index is to compare the actual prediction error with the data of the data set itself. When the numerator and denominator in the formula are divided by the number of predicted samples at the same time, the numerator part becomes the error index introduced earlier. The mean square error, while the denominator part becomes the variance of the predicted sample data set. When the prediction error is smaller and the numerator part is closer to 0, the calculation result of the indicator is closer to 0, that is, the better the prediction effect of the model, When the numerator is closer to the denominator, each predicted value is in a state closer to the mean value, and the final calculated value of the indicator is close to 0, and the prediction effect of the model at this time is extremely poor. Therefore, the value range of this index is between 0 and 1. The larger the value, the better the prediction effect of the model. Generally speaking, when the calculation result of this index exceeds 0.8, the prediction effect of the model is relatively good.

7. Comparison of prediction performance evaluation indicators

Comparing the above six predictive performance evaluation indicators, we can see that when evaluating the predictive performance of a single model, the error between the predicted value of the model and the real value of the predicted sample is mainly used for calculation, and the error between them is different. The form evaluates the predictive performance of the model, such as mean square error, root mean square error, and degree of fit. The absolute percentage error and the symmetric mean absolute percentage error use the absolute value of the error between the predicted value and the true value of the sample to express its predictive performance. In addition, there are certain similarities between these predictive performance evaluation indexes. Except for the degree of fitting, the value ranges of the other five predictive performance evaluation indexes are all between 0 and positive infinity, and the smaller the value, the better the model’s prediction. The better the performance, the better the fitting degree can also be regarded as a deformation of the mean square error, so no matter which prediction performance evaluation index is used to evaluate the prediction effect of the model, it is reasonable to a certain extent.