1. Model Classification Objectives

The classification of the data is two types of positive examples (Postive) and negative examples (Negtive), which are represented by P and N respectively.

At the same time, in the case of prediction, the classification is correctly represented as T (True), and the error is represented as F (False); then there are the following four types of representations:

TP:(True Positive

A correct judgment is a positive example

actual positive example)

TN:(True Negtive

A correct judgment is a negative example

actual negative case)

FP:(Fasle Positive

false positives _

actual negative case)

FN :(False negatives

Negative example _

actual positive example)

Ps. Among them, P and N are the positive and negative examples that the classifier considers. The actual positive and negative examples should be combined with whether the classification is correct.

2. Classification evaluation indicators

①Precision

( The proportion of TP in all judged positive cases )

Precision = TP / ( TP +FP )

② Recall rate (Recall)

( The proportion of TP in the actual positive examples )

Recall = TP / ( TP + FN ) = TP / P

③F1 Score

$F_1=2*\frac{P*R}{P+R}$ P = Precision R = Recall

④support

Support degree: the number of the original real data belonging to this class

⑤Accuracy (Acc)

The proportion of the total predicted as the correct class

$accuracy=\frac{TP+TN}{P+N}$

⑥ weighted average (weighted avg)

weighted avg = $\frac{(P1*support1+P2*support2)}{support1+support2}$

The weight is the support of the sample

3. Target detection and evaluation indicators

①IoU

Intersection and union ratio , that is, the ratio of the intersection and union of the prediction frame (prediction) and the actual target (ground truth). It is used to determine the confidence of the prediction target . When the IoU of a prediction frame is greater than a certain threshold , it is considered a positive classification (usually 0.5 or 0.3)

That is, $IoU=\frac{A\cap B}{A\cup B}$ the popular understanding is the ratio of the overlapping area to the total area

②AP

Average Precision (Average Precision), which means the area under the PR curve. The PR curve is the curve with Precision and Recall as the XY axis respectively . The method of obtaining it is as follows:

1. Arrange the model prediction results in descending order according to the confidence of the predicted value

2. Select 11 different sets of data according to Recall (Recall = [0,0.1...0.9,1.0])

3. Draw the image with these 11 sets of data

and $AP=\frac{1}{11} \sum P_{interp}(r)$

In the case of target detection, Precision and Recall can be simplified to the following formulas:

$Precision=\frac{TP}{all \quad detections}$ $Recall=\frac{TP}{all \quad ground \quad truths}$

Take the image below as an example (green is ground truth, red is prediction)

1. When processing, arrange the prediction frames according to the confidence level from high to low, and draw the following table (where ACC means accumulation)

For example, for object K, ACC TP = the number of all TPs in front of K (including K), and ACC FP is the same; while the formula for solving Precision and Recall of object K can be simplified as:

$Precision=\frac{ACC \quad TP}{ACC \quad TP+ACC \quad FP}=\frac{6}{6+11}$

$Recall=\frac{TP}{all \quad ground \quad truths}=\frac{6}{15}$ (15 is the actual number of targets)

2. After the processing is completed, draw all the points as a PR map

3. Calculate AP

AP = A1+A2+A3+A4, for each group of Recall, take the one with the largest P value for calculation, for example, graph A3=(0.4-0.13333)x0.4285=0.1142

③mAP

The AP of all categories is summed and averaged

mAP is only used as a relatively good metric to compare the difference between the old and new target detection algorithms.

3. Code implementation

See: mAP drawing , the following two txt files are required as data sources when drawing:

detection-results: the coordinates of the predicted results (the first dimension is confidence)

ground-truth: the coordinates of the real target

You can write your own program to convert the xml file of the VOC library to ground-truth and output the prediction results as detection-results

If you need to draw a mixed mAP map of all classes, you need to modify

Modify the cycle, draw all classes in the same graph in turn, and finally save it, which involves clearing the plt control (plt.cla) and saving the image (plt.savefig), see the application of the plt control for details

[Machine Vision] Target Detection Evaluation Index and Its Realization