Machine Learning Metric Calculation

1. As shown in the figure below

1. accuracy

$$\text{ accuracy }=\frac{TP+TN}{TP+TN+FP+FN}$$Accuracy refers to the ratio of the number of correctly predicted samples to the total number of predicted samples. It does not consider whether the predicted samples are positive or negative, but considers all samples.

2. precision (precision rate)

$$\text{ precision }=\frac{TP}{TP+FP}$$

Precision refers to the ratio of the number of correctly predicted positive samples to the number of all predicted positive samples, that is to say, how many of all predicted positive samples are true positive samples. From this we can see that precision only pays attention to the part that is predicted to be a positive sample.

3. recall (recall rate)

$$\text{ recall }=\frac{TP}{TP+FN}$$

It refers to the ratio of the number of correctly predicted positive samples to the total number of real positive samples, that is, how many positive samples I can correctly find out from these samples.

4. F-score

$$F-\text{ score }=\frac{2}{1/\text{ precision }+1/\text{ recall }}$$F-score is equivalent to the harmonic average of precision and recall, and the intention is to refer to two indicators. We can see from the formula that if any value of recall and precision decreases, the F-score will decrease, and vice versa.

5. specificity

$$\text{ specificity }=\frac{TN}{TN+FP}$$The specificity indicator is not often seen. It is relative to sensitivity (recall), which refers to the ratio of the number of correctly predicted negative samples to the total number of real negative samples, that is, how many of them can I correctly find out from these samples? Negative samples.

6. sensitivity(TPR)

$$\text{ sensitivity }=\frac{TP}{TP+FN}=\text{ recall }$$

7. PR curve

We set the vertical axis to precision, the horizontal axis to recall, and change the threshold to obtain a series of pairs and draw a curve. For the prediction effect of different models on the same data set, we can draw a series of PR curves. In general, if one curve completely "surrounds" the other, we can consider that model to classify better than the comparison model.

As shown in the figure below:

Indicators under sample imbalance

background:

In most cases, the classification costs of different categories are not equal, that is, the cost of classifying samples as positive or negative examples cannot be compared. For example, in spam filtering, we hope that important emails will never be misjudged as spam, and in cancer detection, we would rather misjudge than miss. In this case, it is not sufficient to use the classification error rate as a metric alone, as such a metric error conceals the fact of how the sample was misclassified. Therefore, in classification, when the importance of a certain category is higher than other categories, you can use Precision and Recall multiple new indicators that are better than the classification error rate.

8. roc(Receiver Operating Characteristic Curve)

Class imbalance often occurs in real data sets, that is, there are many more negative samples than positive samples (or vice versa), and the distribution of positive and negative samples in the test data may also change over time. In this case, the ROC curve can remain unchanged. At the same time, we can assert that the closer the ROC curve is to the upper left corner, the better the performance of the classifier, which means that the classifier obtains a high true positive rate while the false positive rate is very low.

The following is an example of a ROC curve:

where the abscissa of the curve is the false positive rate (False Positive Rate, FPR), N is the number of real negative samples, and FP is the positive sample predicted by the classifier among the N negative samples The number of , P is the number of real samples. where $FPR=\frac{FP}{FP+TN}$,$TPR=\frac{TP}{TP+FN}$。

For example, if there are 20 samples for 2 classifications, the classification results are as follows:

Now we specify a threshold of 0.9, then only the first sample (0.9) will be classified as a positive example, and all other samples will be classified as Classified as a negative example, therefore, for the threshold of 0.9, we can calculate that the FPR is 0 and the TPR is 0.1 (because there are a total of 10 positive samples, and the number of correct predictions is 1), then we know that there must be a point on the curve is (0, 0.1). Select different thresholds (or "truncation points") in turn, draw all the key points, and then connect the key points to finally get the ROC curve as shown in the figure below.

In fact, there is a more intuitive way to draw the ROC curve, which is to set the scale interval of the horizontal axis to $\frac{1}{N}$, the scale interval of the vertical axis is set to $\frac{1}{P}$, N, P are the number of negative samples and positive samples respectively. Then arrange in descending order according to the output results of the model, traverse the samples in turn, and draw the ROC curve starting from 0. Whenever a positive sample is encountered, a curve with a scale interval is drawn along the vertical axis, and every time a negative sample is encountered, it is along the horizontal axis. Draw a curve with a scale interval. After traversing all the sample points, the curve is drawn.

ROC curve drawing using sklearn:

>>> from sklearnimport metrics
>>> import numpy as np
>>> y = np.array([1, 1, 2, 2]) #假设4个样本
>>> scores = np.array([0.1, 0.4, 0.35, 0.8])
>>> fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2)
>>> fpr #假阳性
array([ 0. ,  0.5,  0.5,  1. ])
>>> tpr #真阳性
array([ 0.5,  0.5,  1. ,  1. ])
>>> thresholds #阈值
array([ 0.8 ,  0.4 ,  0.35,  0.1 ])
>>> #auc(后面会说)
>>> auc = auc = metrics.auc(fpr, tpr)
>>> auc
0.75

Plot the curve:

import matplotlib.pyplot as plt
plt.figure()
lw = 2
plt.plot(fpr, tpr, color='darkorange',
         lw=lw, label='ROC curve (area = %0.2f)' % auc)
plt.plot([0, 1], [0, 1], color='navy', lw=lw, linestyle='--')
plt.xlim([0.0, 1.0])
plt.ylim([0.0, 1.05])
plt.xlabel('False Positive Rate')
plt.ylabel('True Positive Rate')
plt.title('Receiver operating characteristic example')
plt.legend(loc="lower right")
plt.show()

The drawn image is as shown in the figure:

9. auc(Area under curve)

auc refers to the area of ​​the calculated roc. The AUC value is a probability value. When you randomly select a positive sample and a negative sample, the probability that the current classification algorithm ranks the positive sample ahead of the negative sample according to the calculated Score value is the AUC value. The larger the AUC value, the greater the current classification. The more likely the algorithm is to rank positive samples ahead of negative samples, the better it can classify.

def AUC(label, pre):
　　"""
　　适用于ｐｙｔｈｏｎ3.0以上版本
   """
　　#计算正样本和负样本的索引，以便索引出之后的概率值
    pos = [i for i in range(len(label)) if label[i] == 1]
    neg = [i for i in range(len(label)) if label[i] == 0]
 
    auc = 0
    for i in pos:
        for j in neg:
            if pre[i] > pre[j]:
                auc += 1
            elif pre[i] == pre[j]:
                auc += 0.5
 
    return auc / (len(pos)*len(neg))
 
 
if __name__ == '__main__':
    label = [1,0,0,0,1,0,1,0]
    pre = [0.9, 0.8, 0.3, 0.1, 0.4, 0.9, 0.66, 0.7]
    print(AUC(label, pre))

Of course, the formula can also be used for calculation: $$AUC=\frac{{\sum }_{i\in \text{ positiveClass }}{\mathrm{rank}}_i-\frac{M(1+M)}{2}}{M\times N}$$

code show as below:

import numpy as np
def auc_calculate(labels,preds,n_bins=100):
    postive_len = sum(labels)
    negative_len = len(labels) - postive_len
    total_case = postive_len * negative_len
    pos_histogram = [0 for _ in range(n_bins)]
    neg_histogram = [0 for _ in range(n_bins)]
    bin_width = 1.0 / n_bins
    for i in range(len(labels)):
        nth_bin = int(preds[i]/bin_width)
        if labels[i]==1:
            pos_histogram[nth_bin] += 1
        else:
            neg_histogram[nth_bin] += 1
    accumulated_neg = 0
    satisfied_pair = 0
    for i in range(n_bins):
        satisfied_pair += (pos_histogram[i]*accumulated_neg + pos_histogram[i]*neg_histogram[i]*0.5)
        accumulated_neg += neg_histogram[i]

    return satisfied_pair / float(total_case)
 
 y = np.array([1,0,0,0,1,0,1,0,])
 pred = np.array([0.9, 0.8, 0.3, 0.1,0.4,0.9,0.66,0.7])
print("----auc is :",auc_calculate(y,pred))

10. AUROC (Area Under the Receiver Operating Characteristic curve)

Most of the time, AUC refers to AUROC, which is a bad practice, AUC has ambiguity (could be any curve), while AUROC has no ambiguity.

The rest are consistent with AUC.

2. Summary of image segmentation indicators

1. pixel accuracy (marked correctly/total number of pixels)

For the sake of explanation, assume the following: a total of $k+1$ class (from$L_0$to $L_k$, which contains an empty class or background), $p_{ij}$Indicates that this belongs to class $i$ but predicted as classjjThe number of pixels for$j\; .$That is,$p_{ii}$Indicates the true positive sample, and $p_{ij},p_{ji}$The representations are interpreted as false positives and false negatives, respectively.

Its calculation formula is as follows: $$PA=\frac{{\sum }_0^k{p}_{ii}}{{\sum }_{i=0}^k{\sum }_{j=0}^k{p}_{ij}}$$
There are a total of k + 1 k+1 in the image$k+1$ ,$P_{ii}$Indicates that the iiclass$i$$The\; number\; of\; pixels\; of\; class\; i$ (the number of pixels correctly classified),$P_{ij}$Indicates that the iiClass$i$$The\; number\; of\; pixels\; of\; class\; j$ (the number of all pixels)
so the ratio represents the proportion of correctly classified pixels to the total number of pixels.

For $As\; far\; as\; PAis$ concerned, the advantage is simplicity! Disadvantages: If a large area of ​​the image is the background and the target is small, even if the entire picture is predicted as the background, there will be a high PA score, so this indicator is not suitable for evaluating the image segmentation effect of small targets.

2. MPA(Mean Pixel Accuracy)

Its calculation formula is as follows: $$MPA=\frac{1}{1+K}\sum _0^k\frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}}$$

Calculate the accuracy of each category and take the mean!

3. MIou(Mean Intersection over Union)

Computes the ratio of the intersection to the union of two sets, in semantic segmentation, the true and predicted values. $$MIorU=\frac{1}{k+1}\sum _{i=0}^k\frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}}$$

4. FWIoU(Frequency Weighted Intersection over Union)

An improvement of MIou, this method can set weights for each class according to its frequency of occurrence: $$FWIoU=\frac{1}{{\sum }_{i=0}^k{\sum }_{j=0}^k{p}_{ij}}\sum _{i=0}^k\frac{p_{ii}}{{\sum }_{j=0}^k{p}_{ij}+{\sum }_{j=0}^k{p}_{ji}-p_{ii}}$$

3. Summary of target detection indicators

The following indicators are mainly used:

$mAP$: $meanAveragePrecision$ , that is, each category$APAverage$ AP
$AP$ :$The\; area\; under\; the\; PR$ curve will be explained in detail later
$PR$曲线: $Precision-Recall$曲线
$Precision:TP/(TP+FP)$
$Recall:TP/(TP+FN)$
$TP:IoU>$The number of detection frames of$0.5$$GroundTruth$ is calculated only once)
$FP:IoU<=0.5$ detection frame, or detect the sameGT GTThe number of redundant detection frames of$GT$
$FN$ : Number of GTs not detected
$IOU$ : Calculate the ratio of the intersection and union of two sets
$NMS$ : Non-Maximum Suppression
$AP$ calculation

To calculate $AP$ , the first thing to calculate is$TP、FP、FN$.

For a single picture, first traverse $ground$ $truth$ object, and then extract the gt gtof a category we want to calculate$gt$ $objects$ , then read the detection frame of this category detected by the detector (don’t care about other categories), and then filter out the confidence score lower than the confidence threshold, and some confidence thresholds are not set. Sort the remaining detection frames according to the confidence score from high to low, and first judge the detection frame with the highest confidence score and$gt$ $bbox$的$iou$ yes or no$iou$嘈值，young$iou$ is greater than the set$The\; iou$ threshold is judged as$TP$ , put this$g{t}_{b}box$ marked as detected (subsequent sameGT GTThe redundant detection frames of$GT$$FP$ , which is why it is first sorted from high to low according to the confidence score, and the detection frame with the highest confidence score first goes to$iou$ threshold comparison, if greater than$iou$ threshold, regarded as$TP$ , subsequent samegt gtThe detection boxes of$g$$t$$FP$ ),$If\; iou$ is less than the threshold, it is$FP$ . The total number of GTs in a certain category in the picture is fixed, subtract the number of TPs, and the rest is the number of FNs

When there are $TP,FP,After\; the\; FN$ value, we can calculate$precision$与$recall$ . Thus calculating$AP$。

In $Before\; VOC2010$ , you only need to select when$Recall>=0,0.1,0.2,...,1$ out of11 11P precision Precision$at\; 11$ pips$Precision$ max, then$AP$ is this$11$个$Average\; valueofPrecision$ .
In$For\; VOC2010$ and later, Recall is required for each different$Recall$ value (including 0 and 1), select it is greater than or equal to theseRecall RecallP precision Precisionat$R$$ec$$a$$ll\; value$$Precision$ maximum value, then calculatePR PRArea under the$PR$$AP$ value.
$COCO$ data set, set multiple$IOU$ Threshold ($0.5-0.95$,$0.05$ is the step size), in eachIOU IOUThere is a certain type of AP AP$under\; the\; I$$O$$U$ threshold$AP$ value, and then find differentIOU IOUAP APat$I$$O$$U\; Threshold$$AP$ average is the final$AP$ value.
$mAP$ calculation

As the name suggests, all classes of $AP$ -value average is$mAP$。

4. Model efficiency measurement

FLOPs(floating point operations)

It is assumed that the implementation of the convolution operation is in the form of a sliding window, and the nonlinear function does not consume computing resources. Then for $FLOPs$为： $$FLOPs=2HW(C_{in}{K}^2+1)C_{out}$$where $H$,$W$与$C_{in}$is the height, width and channel number of input features, $K$ is the width and length of the convolution kernel,$C_{out}$is the number of output channels. At the same time, it is assumed that the size of the input and output is the same.

For fully connected layers: $$FLOPs=(2I\_-1)O$$ ,$I$ is the dimension of the input,$O$ is the output dimension.