Common evaluation indicators for target detection and their calculation methods

1. Common evaluation indicators for target detection

2. Speed ​​indicator

• Front pass time (ms): The time consumed from inputting an image to outputting the final result, including pre-processing time (such as image normalization), network front pass Time-consuming, post-processing time-consuming (such as non-maximum suppression);
• Frames Per Second (FPS): The number of images that can be processed per second;
• Floating point operations (FLOPS): The number of floating point operations required to process an image has nothing to do with the specific software and hardware. It can fairly compare the performance of different algorithms. Detection speed.

3. Accuracy index

1. Confusion matrix

Note: T/F represents the correctness of the prediction, and P/N represents the result of the prediction.

• TP (True Positive): Indicates the number of times it is actually a positive example and the judgment is also a positive example, that is, it means the number of times it is judged to be a positive example and the judgment is correct;
• FP (False Positive): Indicates the number of times it is actually a negative example but is judged to be a positive example, that is, it means the number of times it is judged to be a positive example but the judgment is wrong;
• TN (True Negative): Indicates the number of times it is actually a negative example and the judgment is also a negative example, that is, it means the number of times it is judged to be a negative example and the judgment is correct;
• FN (False Negative): Indicates the number of times it is actually a positive example but is judged to be a negative example, that is, it means the number of times it is judged to be a negative example but the judgment is wrong;

2.Precision、Recall、F1、Fβ

（1）Precision

Precision: represents the ratio of the number of times that a positive example is correctly determined to the number of times that all positive examples are determined to be positive, and is used to evaluate whether the prediction is accurate;

（2）Recall

Recall rate Recall: represents the ratio of the number of times that a positive example is correctly determined to the number of times that it is actually a positive example. It is used to evaluate the incompleteness of the found example;

（3）F1

F1 is the harmonic average of Precision and Recall;

（4）Fb

Fβ is the weighted harmonic average of Precision and Recall

3IoU(Intersection over Union)

Note: The red box represents the detection result, and the black box represents the true value.

The IoU indicator can be used to measure the accuracy of the external rectangular frame;

4.FAR, FRR, FAR-FRR curve

（1）FAR

FAR (False Acceptance Rate) is the false acceptance rate, also called the false recognition rate, which represents the ratio of the number of times that a positive example is incorrectly determined to the number of times that it is actually a negative example.

（2）FRR

FRR (False Rejection Rate) is the false rejection rate, also called the rejection rate, which represents the ratio of the number of times that a negative example is incorrectly determined to the number of times that it is actually a positive example.

(3) FAR-FRR curve

It reflects the balance relationship between FRR (rejection rate) and FAR (false recognition rate) of the recognition algorithm at different thresholds. The abscissa is FRR (rejection rate), the ordinate is FAR (false recognition rate), and the equal-error rate (EER Equal-Error Rate) is a balance point between the rejection rate and the error rate. The equal-error rate can be obtained The lower the value, the better the performance of the algorithm.

5.TPR, FPR, ROC curve, AUC

（1）TPR

True Positive Rate: true examples/all positive examples;

(2) FPR

False Positive Rate: False positives/all negatives

(3) ROC curve

The ROC (receiver operating characteristic curve) curve is the "receiver operating characteristic" curve, which is a widely accepted system evaluation indicator. Taking FPR as the abscissa and TPR as the ordinate, put (FPR, TPR) corresponding to each threshold into the coordinate system. Connect all the points with a line - this is the ROC curve.

• Threshold selection: Generally speaking, if a point is closer to the upper left corner (the larger the TPR, the smaller the FPR), then the prediction effect of the model is better. If we can reach the upper left corner point (0,1), that will be the most perfect result. When selecting the optimal threshold in practical applications, it is also necessary to consider the requirements for true positives and the tolerance for false positives.
• Model comparison: The ROC curve can also be used to compare different models. There are three ROC curves in the picture on the right. Model A is better than both B and C.
  

（4）AUC

AUC (Area Under Curve): The area under the ROC curve. AUC can also be used to judge the quality of a classifier. Generally speaking, the larger the AUC value, the better.

6.P-R curve, AP, mAP

(1) P-R curve

Selecting different IOU values ​​[0, 1] will result in different Recall and Precision values. The curve formed by using Recall as the independent variable and Precision as the dependent variable is called the P-R curve;

(2)AP and mAP

The area enclosed by the P-R curve and the coordinate axis is the AP value of the current category. AP (Average Precision) is used to evaluate the quality of the model in each category; mAP is the average value of AP, used for Evaluate how well the model performs across all categories. 0<mAP<1, the larger the mAP value, the better the model detection effect.

Note: N is the total number of defect categories

(3) Relationship between P-R curve and ROC

• For a given data set containing positive and negative samples, there is a one-to-one correspondence between ROC space and PR space.
• The ROC curve is mainly used when the distribution of samples in the test set is relatively uniform, and when the distribution of positive and negative samples in the test set changes, the ROC curve can remain unchanged. The consistency of ROC in the face of imbalanced data shows that it can measure the predictive ability of a model itself, which has nothing to do with the positive and negative proportion of samples. However, this insensitive characteristic makes it difficult to see the prediction performance of a model when it faces a change in sample proportion. At this time, the PR curve can better reflect the performance of the classifier.