[CV] From classification to regression: common algorithm evaluation indicators, such as ROC, MAP, etc.

classification problem

Accuracy

Accuracy is one of the simplest and most commonly used classification metrics. The accuracy rate refers to the proportion of the number of samples correctly predicted to the total number of samples. The higher the accuracy rate, the better the classification performance of the model.
$$Accuracy=\frac{TP+TN}{TP+FP+TN+FN}$$
Among them, $TP$ represents the true example, that is, the number of samples that are actually positive and predicted to be positive;$TN$ represents the true negative example, that is, the number of samples that are actually negative and predicted to be negative;$FP$ represents false positives, that is, the number of samples that are actually negative but predicted to be positive;$FN$ represents false negatives, that is, the number of samples that are actually positive but predicted to be negative.

Precision

The precision rate is used to measure the proportion of the samples that the model predicts are positive, that is, to measure the accuracy of the model. The higher the precision rate, the higher the proportion of the samples predicted by the model as the positive class are actually positive.
$$Precision=\frac{TP}{TP+FP}$$

Recall rate or true rate (Recall)

The recall rate is used to measure the proportion of all samples that are actually positive that are predicted to be positive by the model (effective alarm, sensitivity), that is, to measure the comprehensiveness of the model. The higher the recall rate, the better the model can find positive samples.
$$Recall=\frac{TP}{TP+FN}$$

False Positive Rate (FPR)

Negative examples predicted as positive examples among all negative examples (false positive, false positive rate):
$$FPR=\frac{FP}{FP+TN}$$,
where FP represents false positives and TN represents true negatives.

Specificity

It refers to the proportion of all samples that are actually negative examples that are correctly predicted as negative examples. Specifically, specificity can be calculated with the following formula:
$Specificity=\frac{TN}{TN+FP}$

ROC curve

ROC curve is a graphical tool to measure the performance of classification models. The horizontal axis of the ROC curve is the false positive rate (FPR), which is 1-specificity, and the vertical axis is the true positive rate (TPR), whose shape can reflect the performance of the model. The area under the ROC curve (AUC) is also a commonly used classification indicator. The higher the AUC, the better the performance of the model. The value range of AUC is between 0.5 and 1, and the closer the AUC is to 1, the better the performance of the classifier.

The ROC curve is drawn by changing the threshold of the classification model (Threshold). The output result of the classification model is generally a continuous real value, and we can use this value as a threshold to divide the samples into positive examples and negative examples. When the threshold is positive infinity, all samples are judged as negative; when the threshold is negative infinity, all samples are judged as positive; when the threshold is 0, all samples are judged as positive. By changing the threshold, we can get different FPR and TPR to draw the ROC curve.

PR curve (precision-recall curve)

For each class, the predicted boxes are ordered from highest to lowest confidence.

Define a series of thresholds $[t_1,t_2,...,t_n]$ , where$t_1$is the confidence of the prediction box with the lowest confidence, $t_n$is the confidence of the predicted box with the highest confidence.

For each threshold $t_i$, put confidence higher than $t_i$The prediction box of is regarded as a positive sample, lower than $t_i$The predicted box is regarded as a negative sample. According to the IoU value between the predicted frame and the real frame, a set of matching positive samples and corresponding IoU values ​​are obtained.

Calculate the precision and recall rate under this threshold, where the precision is equal to the confidence of the matched predicted box, and the recall rate is the number of positive samples divided by the total number of real boxes.

Use the precision and recall under all thresholds to draw the precision-recall curve, that is, the precision is the horizontal axis, the recall is the vertical axis, and the coordinates of each point are ( P i , R i ) ( $(P_i,R_i)$ , of which$P_i$The recall rate is greater than or equal to $t_i$Accuracy when $R_i$The recall rate is greater than or equal to $t_i$The recall rate when .

AP value

AP is a measure of the area under the precision-recall curve, which represents the average performance of the model at all recall rates. When calculating AP, we need to integrate the area under the curve according to the shape of the precision-recall curve. The value range of AP is between 0 and 1, and the closer to 1, the better the performance of the model.

MAP indicator

Let's take the target detection task as an example. Suppose our model wants to detect objects in a picture, which contains three kinds of fruits: apples, oranges, and bananas. After we used the model to detect this picture, we got the result.

First, we sort all the detection results from high to low confidence, and get the following results:
Category Confidence Bounding Box
Apple 0.8 (10, 10, 100, 100)
Orange 0.7 (20, 20, 120, 120)
Apple 0.6 (50, 50, 150, 150)
Banana 0.5 (30, 30, 130, 130)
Orange 0.4 (60, 60, 160, 160)
Then, we follow the steps above to calculate the AP value for each category. Taking the apple category as an example, we can get the following precision-recall table:
confidence threshold precision-recall rate
0.4 0.5 1.0
0.6 0.67 0.67
0.8 1.0 0.67
According to this table, we can draw the precision-recall curve of the apple category and calculate its AP value. In the same way, we can calculate the AP values ​​for the orange and banana categories. Finally, the final mAP value can be obtained by weighting the AP values ​​of all categories.
When drawing the AP curve, we need to draw the precision-recall curves of all categories in the same graph for comparison. At the same time, we can also draw AP curves under different IoU thresholds to evaluate the performance of the model under different IoU thresholds.

Computations in the COCO dataset

In the COCO dataset, the Map metric is widely used in the evaluation of object detection and segmentation tasks. The COCO data set is a large data set containing more than 330,000 labeled boxes in 80 different categories, so the evaluation of target detection and segmentation tasks on this data set is very representative.
The calculation method of the Map index in the COCO dataset is slightly different from the general Map index calculation method. The specific formula is as follows:
$$AP=\frac{1}{|R|}\sum _{i=1}^{|R|}{P}_i\Delta r_i$$
$$Map=\frac{1}{N}\sum _{i=1}^{N}A{P}_i$$
Among them, $R$ is the set of all real annotation boxes,$N$ is the number of all test pictures. $A{P}_i$Indicates the $The\; average\; precision\; of\; i$ test pictures is obtained by calculating the average of the average precision of all object categories in the picture.$Mapindicator$ .$\_$$P_i$Indicates the $The\; accuracy\; of\; all\; object\; categories\; in\; i$ test pictures,$\Delta r_i$Indicates the iiThe difference in recall for each class among the$i\; test\; images.$

For each test picture, it is necessary to calculate the average precision (AP) of each object category in the picture, and then use the average of the average precision of all object categories as the average precision of the picture. Finally, the average accuracy of all test pictures is averaged, which is the Map indicator of the COCO dataset.
When calculating the average precision for each object category, the difference in recall needs to be considered, that is, objects of different sizes contribute differently to the recall. Therefore, the Map indicator in the COCO dataset will assign different weights to objects of different sizes to more accurately evaluate the model's ability to detect and segment objects of different sizes.
In practical applications, you can use some open source computing tools, such as COCO API and mAP calculator, to easily calculate the Map indicators in the COCO dataset.

F1 value (F1-Score)

The F1 value is the harmonic mean of precision and recall, which is used to measure the overall performance of the model. The higher the F1 value, the better the overall performance of the model.
$$Q1\_=2\frac{PrecisionRecall}{Precision+Recall}$$

The harmonic mean (Harmonic Mean) is a kind of mean in statistics, and its calculation method is the reciprocal of the mean of the reciprocals of n numbers. Specifically, n positive numbers $x_1,x_2,...,x_n$The harmonic mean of can be expressed as:
$$H=\frac{n}{\frac{1}{x_1}+\frac{1}{x_2}+...+\frac{1}{x_n}}=\frac{n}{{\sum }_{i=1}^n\frac{1}{x_i}}$$
Unlike common averages such as the arithmetic mean, geometric mean, and median, the harmonic mean pays more attention to small values, because the reciprocal of any number less than 1 is greater than 1, so when there are individual numbers in a set of data that are small, the harmonic mean will be affected by these decimals and become smaller. Therefore, the harmonic mean is often used in some occasions that need to consider extreme values, such as calculation speed, average rate, etc.

When to use PR curve and when to use ROC curve

PR curve and ROC curve are commonly used classification model performance evaluation indicators, but they are applicable to different scenarios.
The PR curve is mainly used when the proportion of positive and negative samples is unbalanced, for example, when the proportion of positive samples is very low. The horizontal axis of the PR curve is the recall rate, and the vertical axis is the precision rate, which reflects the classification performance of the model under different recall rates. The advantage of the PR curve is that it can better reflect the classification ability of positive samples, especially when the proportion of positive samples is low, it can reflect the performance difference of the model.

The ROC curve is suitable for situations where the proportion of positive and negative samples is relatively balanced. The horizontal axis of the ROC curve is the false positive rate, and the vertical axis is the true positive rate, which reflects the classification performance of the model under different thresholds. In the ROC curve, the point (0,0) means that all samples are predicted as negative samples, and the point (1,1) means that all samples are predicted as positive samples. The closer the curve is to the upper left corner, the better the performance of the model.

In general, when the ratio of positive and negative samples is unbalanced, we can use the PR curve for model performance evaluation; when the ratio of positive and negative samples is relatively balanced, we can use the ROC curve for model performance evaluation.

confusion matrix

The confusion matrix is ​​a matrix that visualizes the classification results, and it can be used to calculate indicators such as accuracy, precision, and recall. The rows of the confusion matrix represent the actual categories, the columns represent the predicted categories, and each element in the matrix represents the number of intersections of the actual and predicted categories.

regression problem

Mean squared error (MSE)
Mean squared error is one of the most commonly used evaluation indicators in regression problems, and it is used to measure the difference between the predicted value of the model and the true value. The smaller the mean square error, the better the predictive performance of the model.
$$MSE=\frac{1}{n}\sum _{i=1}^n(y_i-\widehat{y_i}{)}^2$$
where,$n$ represents the number of samples,$y_i$Indicates the $The\; true\; value\; of\; i$ samples,$\widehat{y_i}$Indicates the predicted value of the model.

Root mean square error (RMSE)
Root mean square error is the square root of the mean square error, which is also used to measure the difference between the model's predicted value and the true value. The smaller the root mean square error, the better the predictive performance of the model.
$$RMSE=\sqrt{\frac{1}{n}\sum _{i=1}^n(y_i-\widehat{y_i}{)}^2}$$

Mean Absolute Error (MAE)
Mean Absolute Error is another commonly used evaluation metric in regression problems, and it is also used to measure the difference between the model's predicted value and the true value. The smaller the mean absolute error, the better the predictive performance of the model.
$$MAE=\frac{1}{n}\sum _{i=1}^n\left|y_i-\widehat{y_i}\right|$$