Regression model evaluation metrics (MAE, MSE, RMSE, R², MAPE)
提示：回归模型简单理解就是：学习模型的因变量（y_predict）是一个连续值。
- Mean Absolute Error (Mean Absolute Error, MAE): It is the average value of the absolute error, which can better reflect the actual situation of the predicted value error.
def MAE(Y_real,Y_pre):#计算MAE
from sklearn.metrics import mean_absolute_error
return mean_absolute_error(Y_real,Y_pre)#Y_real为实际值，Y_pre为预测值
- Mean Square Error (Mean Square Error, MSE): It is the square of the difference between the real value and the predicted value, and then the average of the summation, which is generally used to detect the deviation between the model's predicted value and the real value
def MSE(Y_real,Y_pre):#计算MSE
from sklearn.metrics import mean_squared_error
return mean_squared_error(Y_real,Y_pre)#Y_real为实际值，Y_pre为预测值
- Root Mean Square Error (Root Mean Square Error, RMSE): That is, the square root of the mean square error, and the root mean square deviation represents the sample standard deviation of the difference between the predicted value and the observed value
def RMSE(Y_real,Y_pre):#计算RMSE
from sklearn.metrics import mean_squared_error
return np.sqrt(mean_squared_error(Y_real,Y_pre))#Y_real为实际值，Y_pre为预测值
- R² (R squared, Coefficient of determination): The coefficient of determination reflects the accuracy of the model fitting the data. Generally, the range of R² is 0 to 1. The closer its value is to 1, the stronger the explanatory power of the variables of the equation to y is, and the model fits the data better
def R2(Y_real,Y_pre):#计算R²
from sklearn.metrics import r2_score
return r2_score(Y_real,Y_pre)#Y_real为实际值，Y_pre为预测值
- Mean Absolute Percentage Error (MAPE): In theory, the smaller the value of MAPE, the better the fitting effect of the prediction model and the better accuracy
def MAPE(Y_real,Y_pre):#计算mape
from sklearn.metrics import mean_absolute_percentage_error
return mean_absolute_percentage_error(Y_real,Y_pre)#Y_real为实际值，Y_pre为预测值
Commonly used evaluation indicators for classification models: Accuracy, Precision, Recall, F1-score, and graphic area AUC:
提示：分类模型简单理解就是：学习模型的因变量（y_predict）是一个离散值（结果只有n个类别），例如：n为3就是3分类问题。
例子：根据肿瘤的大小（自变量：Tumor Size）来预测肿瘤（因变量：Malignant）是恶性样本（negative examples）还是良性样本（positive examples）。（二分类问题）
-
Basic indicators: error rate (the proportion of misclassified samples to the total samples)
-
Basic indicators: accuracy rate (the proportion of correctly classified samples to the total samples)
-
Confusion matrix (two-category problem): Under the classification task, there are four different combinations between the predicted result (Predicted Condition) and the correct label (True Condition), forming a confusion matrix
-
Precision rate Precision: the proportion of positive samples in which the prediction result is positive (also known as precision)
-
Recall rate (recall rate) Recall: the proportion of the predicted results in the samples that are actually positive samples (recall is complete, the ability to distinguish positive samples)
分类问题评测指标API:
#分类问题评测指标API
from sklearn.metrics import classification_report
#y_test为实际值，y_predict为预测值
#labels:指定类别对应的数字,target_names：⽬标类别名称
ret = classification_report(y_test, y_predict, labels=(2,4), target_names=("良性", "恶性"))
# 打印返回的ret：包括每个类别精确率（Precision）与召回率（Recall）
print(ret)
-
There are other evaluation criteria: F1-score (reflects the robustness of the model)
-
TPR and FPR
-
ROC curve and AOC index
-
Background: In order to avoid evaluation problems under sample imbalance,
1.例子：
if 99 samples are cancer and 1 sample is non-cancer, I will directly predict all positive cases (cancer is the default case), and the accuracy rate is 99%,
but the effect is not good.
2.问题：
This prediction model (a model that directly predicts all positive examples) is only 99% accurate under the current unbalanced data set, and the results under other data sets are unknown. In some important businesses, such a model is not universal at all, which is terrible.
3.解决：
Use the ROC curve -
Meaning: The horizontal axis of the ROC curve is FPRate, and the vertical axis is TPRate. When the two are equal, the meaning is: for samples regardless of whether the real category is 1 or 0, the probability of the classifier predicting 1 is equal. When the AUC is 0.5.
-
Graphic:
-
AUC indicator
1. The probability meaning of AUC is to randomly select a pair of positive and negative samples, and the probability that the positive sample score is greater than the negative sample score 2. The
range of AUC is between [0, 1], and the closer to 1, the better, the closer 0.5 is random guessing
3. AUC=1, a perfect classifier, when using this prediction model, no matter what threshold is set, a perfect prediction can be obtained. But for most forecasting occasions, there is no perfect classifier.
4. When 0.5<AUC<1, it is better than random guessing. If the classifier (model) properly sets the appropriate threshold, it can have predictive value. -
AUC calculation API
- ROC curve drawing
- drawing process
- 1. Build a model and sort the probability values of the model from large to small
- 2. Start to take the value from the point with the highest probability, carry out the calculation of tpr and fpr all the time, and then build the overall model to get the result
- 3. In fact, it is solving the integral (area)
- Case background
Assume that there are 6 samples, two of which are positive samples, and a sample sequence (1:1, 2:0, 3:1, 4:0, 5:0, 6:0) is obtained, and the front indicates the serial number , the latter ones represent positive samples (1) or negative samples (0).
Then the probability sequence of the positive samples is calculated by the model in these 6 samples. - three conditions
-
1. If the probability sequence is (1:0.9, 2:0.7, 3:0.8, 4:0.6, 5:0.5, 6:0.4)
together with the original sequence, get the sequence (from high to low probability)
The drawing steps are:
1) Sort the probability sequence from high to low to get the sequence (1:0.9, 3:0.8, 2:0.7, 4:0.6, 5:0.5, 6:0.4); 2)
From the highest probability Start to take a point as the positive class, take point 1, and calculate TPR=0.5, FPR=0.0;
3) Start with the highest probability, then take another point as the positive class, take point 3, and calculate TPR= 1.0, FPR=0.0;
4) Then take a point from the largest point as the positive class, take point 2, and calculate TPR=1.0, FPR=0.25;
5) By analogy, 6 pairs of TPR and FPR are obtained.
Then these 6 pairs of data are composed of 6 points (0,0.5), (0,1.0), (0.25,1), (0.5,1), (0.75,1), (1.0,1.0).
These 6 points can be drawn in the two-dimensional coordinate system.
As shown in the figure is the ROC curve. -
2. If the probability sequence is (1:0.9, 2:0.8, 3:0.7, 4:0.6, 5:0.5, 6:0.4)
together with the original sequence, get the sequence (from high to low probability)
The drawing steps are:
1) Sort the probability sequence from high to low to get the sequence (1:0.9, 2:0.8, 3:0.7, 4:0.6, 5:0.5, 6:0.4)
; Start to take a point as the positive class, take point 1, and calculate TPR=0.5, FPR=0.0;
3) Start with the highest probability, then take another point as the positive class, take point 2, and calculate TPR= 0.5, FPR=0.25;
4) Then take a point from the maximum as the positive class, take point 3, and calculate TPR=1.0, FPR=0.25; 5) By analogy,
6 pairs of TPR and FPR are obtained.
Then these 6 pairs of data are composed of 6 points (0,0.5), (0.25,0.5), (0.25,1), (0.5,1), (0.75,1), (1.0,1.0).
These 6 points can be drawn in the two-dimensional coordinate system.
-
3. If the probability sequence is (1:0.4, 2:0.6, 3:0.5, 4:0.7, 5:0.8, 6:0.9)
together with the original sequence, get the sequence (from high to low probability)
The drawing steps are:
1) Sort the probability sequence from high to low to get the order (6:0.9,5:0.8,4:0.7,2:0.6,3:0.5,1:0.4)
; Start to take a point as the positive class, take point 6, and calculate TPR=0.0, FPR=0.25;
3) Start with the highest probability, then take another point as the positive class, take point 5, and calculate TPR= 0.0, FPR=0.5;
4) Then take a point from the largest point as the positive class, take point 4, and calculate TPR=0.0, FPR=0.75;
5) By analogy, 6 pairs of TPR and FPR are obtained.
Then these 6 pairs of data are composed of 6 points (0.25,0.0), (0.5,0.0), (0.75,0.0), (1.0,0.0), (1.0,0.5), (1.0,1.0).
These 6 points can be drawn in the two-dimensional coordinate system.
-
- meaning interpretation
- As shown in the example above, there are a total of 6 points, 2 positive samples, and 4 negative samples. There are 8 situations in which one positive sample and one negative sample are taken.
- The above
第⼀种情况is taken from top to bottom. No matter how you choose, the probability of positive samples is always higher than that of negative samples, so the probability of pairing is 1, and AUC=1. Look at the ROC curve again, what is its integral? Also 1, the integral of the ROC curve is equal to the AUC. - In the above
第⼆种情况, if samples 2 and 3 are obtained, the classification is wrong, and the classification is correct in other cases; so the probability of classification is 0.875, AUC=0.875. Looking at the ROC curve again, its integral is also 0.875, and the integral of the ROC curve is equal to AUC. - The above
第三种情况, no matter how you choose, is wrong, so the probability of being right is 0, AUC=0.0. Looking at the ROC curve again, its integral is also 0.0, and the integral of the ROC curve is equal to AUC. - AUC means - Area Under roc Curve, that is
ROC曲线的积分, it is also under the ROC curve⾯积. - The drawing
ROC曲线的意义is obvious, and the possible misclassifications are constantly deducted. Every time a negative sample is taken from the point with the highest probability, it will cause all the positive samples below it to be misclassified. Therefore, it is necessary to Subtract the number of positive samples below it (1-TPR, the proportion of the remaining positive samples). After the overall ROC curve is drawn, the AUC is determined, and the probability of pairing can also be calculated.