Table of contents
Precision (Precision) and recall (Recall)
PR curve (Precision-Recall Curve) and AP
Object detection is an important task in the field of computer vision, and its goal is to identify the location and category of objects in images or videos. In order to evaluate the performance of an object detection algorithm, a series of metrics are used to quantify the model's accuracy, recall, precision, and ability to handle different categories. This article will introduce common target detection performance indicators in detail, including precision rate, recall rate, F1 score, IoU, AP, mAP, PR curve, etc., and provide relevant formulas and cases.
Precision (Precision) and recall (Recall)
Precision and recall are important indicators for evaluating the performance of object detection models, and they are usually used in binary classification tasks. In object detection, precision represents the ratio between the number of objects correctly identified by the model and the number of all boxes predicted by the model to be objects. Recall represents the ratio of the number of objects correctly identified by the model to the total number of objects.
The formula is as follows:
Accuracy: Acc = ( TP + TN ) / ( P +N )
Accuracy ( precision ): TP / ( TP+FP ) = TP / P
Recall rate (recall): TP / (TP + FN ) = TP / T
Example: Suppose we have an object detection model tested on a set of images involving 10 object objects. The model identified 8 objects, 6 of which were real (true positives), 2 were misidentified (false positives), and 2 were actually not recognized (false negatives). Then, the precision is 6 / (6 + 2) = 0.75 and the recall is 6 / (6 + 2) = 0.75.
F1 score (F1 Score)
The F1 score is the harmonic mean of precision and recall, which is used to comprehensively consider the accuracy and recall of the model. The F1 score is useful for dealing with imbalanced datasets or when there is a tradeoff between precision and recall.
The formula is as follows:
Case: Suppose an object detection model has a precision rate of 0.8 and a recall rate of 0.7. Then, the F1 score is 2 * (0.8 * 0.7) / (0.8 + 0.7) = 0.7619.
IoU（Intersection over Union）
IoU is a measure of the degree of overlap between the predicted bounding box and the true bounding box, and is often used to evaluate the quality of the object detection box. IoU is measured by calculating the intersection area of the predicted box and the ground truth box divided by their union area.
The formula is as follows:
Typically, a predicted box is considered a correct prediction if the IoU is greater than a certain threshold (e.g. 0.5).
Case: Consider a target detection task, the position of the real frame is [20, 30, 100, 150], and the position of the predicted frame is [25, 35, 95, 145]. The intersection area is (95 - 25) * (145 - 35) = 6000, the real box area is (100 - 20) * (150 - 30) = 9000, and the predicted box area is (95 - 25) * (145 - 35) = 6000. The union area is 9000 + 6000 - 6000 = 9000. Therefore, the IoU is 6000 / 9000 = 0.6667.
PR curve (Precision-Recall Curve) and AP
A PR curve is a graphical representation of the accuracy of a model at different recall rates. In target detection, the recall rate refers to the ratio of the number of correctly detected positive samples to the number of all actual positive samples, while the precision rate refers to the ratio of the number of correctly detected positive samples to the number of samples predicted as positive samples by the model. .
A simple example is used to demonstrate the calculation of average precision (AP). Suppose there are a total of 5 apples in the dataset. We collect all predictions made by the model for apples and rank them according to their confidence level (from highest to lowest). The second column indicates whether the prediction was correct or not. It is correct if it matches the ground truth and loU ≥ 0.5.
In the table, the Rank column indicates the serial number of the boxes arranged from high to low in confidence, correct indicates whether the box is correct, Precision indicates the calculated precision rate, and Recall indicates the calculated recall rate.
The calculation of precision and recall is a cumulative process, not independent of each box.
For example:
In the first box, precision rate = 1/1 = 1.0, recall rate = 1/5 = 0.2;
At the second box, precision = 2/2 = 1.0, recall = 2/5 = 0.4;
At the third box, precision = 2/3 = 0.67, recall = 2/5 = 0.4;
...
At this time, we can draw the PR curve according to the precision rate and recall rate:
AP is the area under the PR curve. The sampling points will be set. Generally, the horizontal axis is divided into 10 or 100 segments on average within the range of 0 to 1. The final sampled values are added and divided by the number of sampling points. Of course there are other ways.
mAP（mean Average Precision）
mAP is the average of all category APs, which is usually used to measure the overall performance of multi-category object detection tasks. It comprehensively evaluates the performance of the model on various categories and can accurately reflect the performance of the model on different categories.
Case: Suppose we have a multi-category target detection model, and the calculated AP for each category is as follows:
| category | AP |
|---|---|
| dog | 0.85 |
| Cat | 0.75 |
| vehicle | 0.90 |
| pedestrian | 0.70 |
Then mAP is (0.85 + 0.75 + 0.90 + 0.70) / 4 = 0.80.
Object detection performance metrics play an important role in evaluating the performance of models on different datasets and tasks. Indicators such as precision rate, recall rate, F1 score, IoU, AP, mAP and PR curve can comprehensively consider the performance of the model in different aspects and help us better understand the advantages and limitations of the model. In practical applications, according to the characteristics and requirements of the task, it is crucial to choose a suitable indicator to evaluate the performance of the model.