Target detection evaluation indicators Precision, Recall, mAP

Target detection evaluation indicators Precision, Recall, mAP

参考：
Object-Detection-Metrics
A Survey on Performance Metrics for Object-Detection Algorithms

1. Important definitions

1.1. Intersection over Union (IOU)

Intersection Over Union (IOU) is used to measure the coincidence rate of two boxes, and its calculation formula is as follows:
$$\mathrm{IOU}=\frac{\mathrm{area}\left(B_p\cap B_{gt}\right)}{\mathrm{area}\left(B_p\cup B_{gt}\right)}$$
where $B_p$is the box predicted by the model, $B_{gt}$For ground truth. Intuitive point:

During the experimental evaluation process, an IOU threshold will be set to judge $B_p$For positive or negative samples, for example, setting the IOU threshold to 0.5 means:

• IOU ≥ 0.5 ：$B_p$is a positive sample;
• IOU < 0.5 ：$B_p$is a negative sample.

Thresholds are often set at 50%, 75% or 95%.

1.2.TP , FP , FN

• True Positive (TP) : A correct detection, ie $B_p$with $B_{gt}$IOU ≥ threshold (initially set IOU threshold);
• False Positive (FP): A wrong detection, representing $B_p$and the truth value $B_{gt}$的IOU < threshold；
• False Negative (FN) : represents the true value $B_{gt}$Not detected, that is, all $B_p$There is no connection with the $B_{gt}$coincident;

1.3.Precision (accuracy rate), Recall (recall rate)

• Precision (accuracy rate): refers to the ratio of correct detection in all boxes predicted by the model on a picture, the expression
  $$\text{ Precision }=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}=\frac{TP}{\text{ all detections }}$$
• Recall (recall rate): refers to the ratio of all ground truths that are correctly matched
  $$\text{ Recall }=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}=\frac{TP}{\text{ all ground truths }}$$

2.Average Precision（AP）

AP is the average value of the Precision corresponding to all recalls between 0 and 1. From the formulas of Precision and Recall, it can be seen that as the model predicts more frames (all detections) on the picture, TP will have an upper limit, so the corresponding Precision will become smaller; when there are more all detections, it means that there are more More ground truth may be correctly matched, that is, TP will increase slightly, and Recall will become larger at this time. The reverse is also true, so we need the detector to maintain a high accuracy rate as the Recall increases (more and more ground truths are correctly matched).

Average Precision (AP) is used to calculate the area of ​​the Precision x Recall curve, as shown in the figure below, by interpolation.

There are examples in the above reference article, here is the analysis:

There are 7 images in total here, where the green bounding boxes represent 15 ground truths and the red bounding boxes represent 24 pre-boxes. Each predicted box pre-box contains a confidence. The following table shows each pre-box and the corresponding confidence, please note:

• As long as the IOU between a pre-box and a ground truth is greater than the set threshold, it is marked as TP, otherwise it is FP.
• On some pictures, if a ground truth overlaps with more than one pre-box, the pre-box with the largest IOU is TP, and the rest are FP.
  

Because both Precision and Recall are calculated by FP and TP, the Precision x Recall curve is also drawn by calculating the cumulative TP or FP detection precision-recall value. First, we need to sort according to the credibility of the detection, and then calculate the Precision and Recall of each cumulative detection, as shown in the table below:

where Acc TP is the cumulative number of TPs. Calculate the current Precision and Recall based on the current Acc TP and Acc FP. Draw a line chart:

As mentioned above, the interpolation method is used to calculate AP. There are two interpolation methods here:

2.1 11-point interpolation

Calculation formula:
AP = 1 11 ∑ r ∈ { 0 , 0.1 , … , 1 } ρ interp ⁡ ( r ) \mathrm{AP}=\frac{1}{11} \sum_{r \in\{0,0.1 , \ldots, 1\}} \rho_{\operatorname{interp}(r)}AP=111​r∈{ 0,0.1,…,1}∑​rinterp ( r )
Enclosure
$$r_{}=\underset{\bar{r}:\tilde{r}\ge r}{\max }r\left(\tilde{r}\right)$$
The meaning of this formula is under 11 levels of recall, namelyr ∈ { 0 , 0.1 , … , 1 } r \in\{0,0.1, \ldots,1\}r∈{ 0,0.1,…,1 } for interpolation. For example:

• when $r=For\; example$ ,$r_{}={\max }_{\bar{r}:\tilde{r}\ge 0}r\left(\tilde{r}\right)$, while at all$\tilde{r}\ge 0$ , can make$r\left(\tilde{r}\right)$the largest$\tilde{r}$ When it is equal to 0.0666, Precision=1 at this time.
• when $r=0.1$时,$r_{}={\max }_{\bar{r}:\tilde{r}\ge 0.1}r\left(\tilde{r}\right)$for all$\tilde{r}\ge$In the point of$0.1$$r\left(\tilde{r}\right)$the largest$\tilde{r}$ When it is equal to 0.1333, Precision=0.6666 at this time.
• And so on, there is no $r>=0.5$ , so it is 0 afterwards, and the following formula is obtained:
  AP = 1 11 ∑ r ∈ { 0 , 0.1 , … , 1 } ρ interp ( r ) AP = 1 11 ( 1 + 0.6666 + 0.4285 + 0.4285 + 0.4285 + 0 + 0 + 0 + 0 + 0 + 0 ) AP = 26.84 % \begin{array}{l} AP=\frac{1}{11} \sum_{r \in\{0,0.1, \ldots, 1\ }} \rho_{\text {interp }(r)} \\ AP=\frac{1}{11}(1+0.6666+0.4285+0.4285+0.4285+0+0+0+0+0+0) \ \ AP=26.84 \% \end{array}AP=111​∑r∈{ 0,0.1,…,1}​rinterp  ( r )AP=111​(1+0.6666+0.4285+0.4285+0.4285+0+0+0+0+0+0)AP=26.84%​

2.1 Full point interpolation

This picture is easy to understand, that is, to calculate the area:


Calculation formula:
$$\begin{array}{l}AP=A_1+A_2+A_3+A_4\\ A_1=(0.0666-0)\times 1=0.0666\\ A_2=(0.1333-0.0666)\times 0.6666=0.04446222\\ A_3=(0.4-0.1333)\times 0.4285=0.11428095\\ A_4=(0.4666-0.4)\times 0.3043=0.02026638\\ AP=0.0666+0.04446222+0.11428095+0.02026638\\ AP=0.24560955\\ AP=24.56\%\end{array}$$