In image classification or image segmentation, in order to evaluate the prediction effect of the model , it is usually necessary to compare the gap between the predicted value and the real label value, that is, the error, during the training process.

Table of contents

Evaluation Metrics for Image Classification Process

correctness/accuracy

Accuracy

recall rate

F1 score

Evaluation Metrics for Image Segmentation Process

confusion matrix

Confusion matrix generation code

IOU与MIOU

IOU calculation code

dice coefficient

Dice coefficient calculation code

The relationship between IOU and dice coefficient

Evaluation Metrics for Image Classification Process

confusion matrix

Confusion matrix, a matrix used to summarize classification results, a square matrix of N*N, where N represents the number of categories. The rows of the confusion matrix represent the true classes and the columns represent the predicted classes.

For example: for a two-category problem, the confusion matrix is:

	predicted value = 1	predicted value = 0
true value = 1	TP	FN
true value = 0	FP	TN

TP is true positive, FP is false positive, FN is false negative, and TN is true negative.

correctness/accuracy

Measures the proportion of the model that correctly classifies the entire dataset

$accuracy=\frac{TP+TN}{TP+FP+TN+FN}$

Accuracy

Among the samples identified as this class, how many are correct

(1) The precision of category 1:

$\frac{TP}{TP+FP}$

(2) The accuracy rate precision of category 0:

$\frac{TN}{TN+FN}$

recall rate

In this type of samples, how many were recovered?

(1) The recall rate recall of category 1:

$\frac{TP}{TP+FN}$

(2) The recall rate recall of category 0:

$\frac{TN}{TN+FP}$

Note: The correct rate refers to all samples; the precision rate or recall rate refers to a certain class , and the precision rate or recall rate of a certain class

F1 score

The harmonic average of precision rate and recall rate can achieve both. The characteristic is: if the two are extremely unbalanced, when one value is extremely high and the other is extremely low, the F1 score obtained is also extremely low; only when both are very high, F1 will be high.

$\frac{2*precision*recall}{precision+recall}$

Among them, precision recall is the above-mentioned precision rate and recall rate respectively, and it also calculates the F1 score for a certain class.

Evaluation Metrics for Image Segmentation Process

confusion matrix

Image segmentation is to classify images pixel by pixel. The predicted value and the real value are no longer a single label value in the classification process (a whole picture represents a category), but a two-dimensional array of h*w (each pixel represents a category). Therefore, you can use the .flatten() function to flatten the predicted value and the real value into a one-dimensional array, and then traverse one by one to accumulate the value of the confusion matrix.

Confusion matrix generation code

"""
class_num:分割类别数
model：模型
inputs:模型输入     shape=(b,c,h,w) b-batchsize c-channel(图片输入通道数)
labels:模型真实标签  shape=(b,h,w)
outputs:模型预测值   shape=(b,class_num,h,w)  class_num维度代表此像素点是所有类别的可能性，outputs.max(dim=1)[1]即可得到最大可能性的类别(预测类别)
"""
import numpy as np
conf_mat=np.zeros((class_num,class_num))
outputs=model(inputs)

pre_label=outputs.max(dim=1)[1].cpu().numpy().flatten()
true_label=labels.cpu().numpy().flatten()

# 混淆矩阵
for i,j in zip(true_label,pre_label):
    conf_mat[i][j]+=1

IOU与MIOU

Iou ( intersection over union) is widely used in semantic segmentation and refers to the prediction accuracy of each category

$\frac{target\cap prediction}{target\cup prediction}$

Among them, target is the number of labels in this category, and prediction is the number of predicted categories

miou (mean intersection and union ratio) is the average of all category iou values

Example 1: Assuming two categories of image segmentation, confusion matrix 2*2

	predicted value = 1	predicted value = 0
true value = 1	TP	FN
true value = 0	FP	TN

(1) IOU of Category 1:

$\frac{TP}{TP+FP+FN}$

At this point, target=TP+FN prediction=TP+FP

(2) IOU of category 0:

$\frac{TN}{TN+FP+FN}$

At this point, target=TN+FP prediction=TN+FN

IOU calculation code

import numpy as np

confusion=np.array([[100,1],[1,200]])  # 一个2*2的混淆矩阵
iou = np.diag(confusion) / (confusion.sum(axis=1) + confusion.sum(axis=0)- np.diag(confusion))
print(iou)    # [0.98039216 0.99009901]

# miou为均交并比
miou=np.mean(iou)
print(miou)   # 0.9852455833818676

dice coefficient

The dice coefficient is a set similarity measurement function, which is usually used to calculate the similarity between two samples , and the value range is between [0,1].

$s=\frac{2|X\cap Y|}{|X|+|Y|}$

For semantic segmentation, X is the GT segmented image (Ground Truth), and Y is the pred segmented image (predicted value)

Example 2: The confusion matrix plot is the same as Example 1

(1) The dice coefficient of category 1:

$s=\frac{2*TP}{2*TP+FN+FP}$

(2) The dice coefficient of category 0:

$s=\frac{2*TN}{2*TN+FN+FP}$

Dice coefficient calculation code

import numpy as np

conf_mat=np.array([[100,1],[1,200]])
dice = 2 * np.diag(conf_mat) / (conf_mat.sum(axis=1) + conf_mat.sum(axis=0))  # 各个类别的dice系数
print(dice)             # [0.99009901 0.99502488]
print(np.mean(dice))    # 0.9925619427614403

The relationship between IOU and dice coefficient

$iou=\frac{dice}{2-dice}$

Therefore, it can be deduced that

$dice=\frac{2*iou}{1+iou}$

Since the iou value is between [0,1], the dice coefficient is greater than or equal to the iou value.

Higher scores can be obtained by using the dice coefficient.

Evaluation indicators for image classification and image segmentation (confusion matrix accuracy rate precision recall rate F1 score IOU dice coefficient)

Evaluation Metrics for Image Classification Process

confusion matrix

correctness/accuracy

Accuracy

recall rate

F1 score

Evaluation Metrics for Image Segmentation Process

confusion matrix

Confusion matrix generation code

IOU与MIOU

IOU calculation code

dice coefficient

Dice coefficient calculation code

The relationship between IOU and dice coefficient

Guess you like