七、模型评估指标

企业开发 2023-01-28 20:08:31 阅读次数: 0

当训练好模型之后，检测模型训练效果如何，评价指标有哪些？通过查阅相关资料，我将以这五个指标来对所训练的模型进行评估，下图是评价指标运行结果图。

在这里插入图片描述

一、混淆矩阵（Confusion Matrix）

解释：也就是个n维矩阵，n表示分类的类别数。
具体的表示如下(这里以二分类任务为例)：也就是图中的二维矩阵
在这里插入图片描述在这里插入图片描述
上述的所有指标都是建立在混淆矩阵的基础上进行计算的
我这里以织物毛球和纹理进行识别，毛球为Positive，纹理为Negative
这个二维矩阵有四个参数：

参数 解释
True Positive 模型预测识别为Positive，识别正确True；实际为Positive
False Negative 模型预测识别为Negative，识别错误False；实际为Positive
False Positive 模型预测识别为Positive，识别错误False；实际为Negative
True Negative 模型预测识别为Negative，识别正确True；实际为Negative

这些值对测试图像中所有像素点进行分类统计

代码实现：

修改：SegmentationMetric(2)改成实际训练模型的分类数，我这个模型训练的是二分类任务
imgPredict和imgLabel 改成自己模型预测的图像和标签图像的路径
实际上，imgLabel为正确答案，依次遍历imgPredict中像素点，与正确答案进行对比，统计上述参数的个数，最后绘制成混淆矩阵。

import numpy as np
import cv2

class SegmentationMetric(object):
    def __init__(self, numClass):
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,) * 2)  # 混淆矩阵（空）


    def addBatch(self, imgPredict, imgLabel):
        assert imgPredict.shape == imgLabel.shape
        self.confusionMatrix += self.genConfusionMatrix(imgPredict, imgLabel)  # 得到混淆矩阵
        return self.confusionMatrix


    def genConfusionMatrix(self, imgPredict, imgLabel): 
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass ** 2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        return confusionMatrix


# 测试内容
if __name__ == '__main__':
    imgPredict = cv2.imread("../result/qqq.png")
    = cv2.imread("../result/img.jpg")

    imgPredict = np.array(cv2.cvtColor(imgPredict, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    imgLabel = np.array(cv2.cvtColor(imgLabel, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)

    metric = SegmentationMetric(2)  # 2表示有2个分类
    ConfusionMatrix = metric.addBatch(imgPredict, imgLabel)

    print('ConfusionMatrix is :\n', ConfusionMatrix)

运行结果如下：在这里插入图片描述

二、像素准确率PA（Pixel Accuracy）

PA最后的输出是一个数值，因为是，无论多少类别的分类，都是跟标准标签进行对比，一致就是True，不一致就是False

PA，别的论文也称为准确率、Acc等，都指的是像素准确率
Accuracy = (TP + TN) / (TP + TN + FP + FN)，也就是对角线元素之和/总的元素之和
（99586+1150）/（99586+1108+556+1150）= 0.983750，这也对应了第一张图的显示结果

代码实现：

import numpy as np
import cv2

class SegmentationMetric(object):
    def __init__(self, numClass):
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,) * 2)  # 混淆矩阵（空）


    def addBatch(self, imgPredict, imgLabel):
        assert imgPredict.shape == imgLabel.shape
        self.confusionMatrix += self.genConfusionMatrix(imgPredict, imgLabel)  # 得到混淆矩阵
        return self.confusionMatrix


    def genConfusionMatrix(self, imgPredict, imgLabel):
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass ** 2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        return confusionMatrix


    def pixelAccuracy(self):
        acc = np.diag(self.confusionMatrix).sum() / self.confusionMatrix.sum()
        return acc

# 测试内容
if __name__ == '__main__':
    imgPredict = cv2.imread("../result/qqq.png")
    imgLabel = cv2.imread("../result/img.jpg")

    imgPredict = np.array(cv2.cvtColor(imgPredict, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    imgLabel = np.array(cv2.cvtColor(imgLabel, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)

    metric = SegmentationMetric(2)  # 2表示有2个分类
    ConfusionMatrix = metric.addBatch(imgPredict, imgLabel)

    PixelAccuracy = metric.pixelAccuracy()

    print('PixelAccuracy is :\n', PixelAccuracy)

运行结果如下：
在这里插入图片描述

三、类别像素准确率CPA（Class Pixel Accuracy）

CPA与PA不同，PA是将整体区分True和False最后结果是一个数值
CPA则先将不同的类别进行划分，每个类别再分别与标签给定的正确答案进行对比统计，最后的个数是类别个数，有几个类别就是几个数值。

代码实现：

import numpy as np
import cv2

class SegmentationMetric(object):
    def __init__(self, numClass):
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,) * 2)  # 混淆矩阵（空）


    def addBatch(self, imgPredict, imgLabel):
        assert imgPredict.shape == imgLabel.shape
        self.confusionMatrix += self.genConfusionMatrix(imgPredict, imgLabel)  # 得到混淆矩阵
        return self.confusionMatrix


    def genConfusionMatrix(self, imgPredict, imgLabel):
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass ** 2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        return confusionMatrix

    def classPixelAccuracy(self):
        classAcc = np.diag(self.confusionMatrix) / self.confusionMatrix.sum(axis=1)
        return classAcc  # 返回的是一个列表值，如：[0.90, 0.80, 0.96]，表示类别1 2 3各类别的预测准确率

# 测试内容
if __name__ == '__main__':
    imgPredict = cv2.imread("../result/qqq.png")
    imgLabel = cv2.imread("../result/img.jpg")

    imgPredict = np.array(cv2.cvtColor(imgPredict, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    imgLabel = np.array(cv2.cvtColor(imgLabel, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)

    metric = SegmentationMetric(2)  # 2表示有2个分类
    ConfusionMatrix = metric.addBatch(imgPredict, imgLabel)

    cpa = metric.classPixelAccuracy()

    print('classPixelAccuracy is :\n', cpa)

因为是二分类任务，故分别显示这两个类别的PA值
运行效果如下：
在这里插入图片描述

四、类别平均像素准确率MPA（Mean class Pixel Accuracy）

也就是将所有的CPA加一块，求个平均值

代码实现：

import numpy as np
import cv2

class SegmentationMetric(object):
    def __init__(self, numClass):
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,) * 2)  # 混淆矩阵（空）


    def addBatch(self, imgPredict, imgLabel):
        assert imgPredict.shape == imgLabel.shape
        self.confusionMatrix += self.genConfusionMatrix(imgPredict, imgLabel)  # 得到混淆矩阵
        return self.confusionMatrix


    def genConfusionMatrix(self, imgPredict, imgLabel):
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass ** 2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        return confusionMatrix

    def classPixelAccuracy(self):
        classAcc = np.diag(self.confusionMatrix) / self.confusionMatrix.sum(axis=1)
        return classAcc  # 返回的是一个列表值，如：[0.90, 0.80, 0.96]，表示类别1 2 3各类别的预测准确率

    def meanPixelAccuracy(self):
        classAcc = self.classPixelAccuracy()
        meanAcc = np.nanmean(classAcc)  # np.nanmean 求平均值，nan表示遇到Nan类型，其值取为0
        return meanAcc  # 返回单个值，如：np.nanmean([0.90, 0.80, 0.96, nan, nan]) = (0.90 + 0.80 + 0.96） / 3 =  0.89

# 测试内容
if __name__ == '__main__':
    imgPredict = cv2.imread("../result/qqq.png")
    imgLabel = cv2.imread("../result/img.jpg")

    imgPredict = np.array(cv2.cvtColor(imgPredict, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    imgLabel = np.array(cv2.cvtColor(imgLabel, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)

    metric = SegmentationMetric(2)  # 2表示有2个分类
    ConfusionMatrix = metric.addBatch(imgPredict, imgLabel)

    mpa = metric.meanPixelAccuracy()

    print('meanPixelAccuracy is :\n', mpa)

（0.98899637+0.67409144）/ 2 = 0.831543905
运行结果如下：
在这里插入图片描述

五、交并比IoU（Intersection Over Union）

通俗来说：将标签图像和模型预测出的图像重叠一下，分别取交集和并集，这里的交集和并集取得是统计像素点的个数
IoU是按不同类别分别进行求解的，几个类别就有几个IoU
IoU = 交集 / 并集
在这里插入图片描述

代码实现：

import numpy as np
import cv2

class SegmentationMetric(object):
    def __init__(self, numClass):
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,) * 2)  # 混淆矩阵（空）


    def addBatch(self, imgPredict, imgLabel):
        assert imgPredict.shape == imgLabel.shape
        self.confusionMatrix += self.genConfusionMatrix(imgPredict, imgLabel)  # 得到混淆矩阵
        return self.confusionMatrix


    def genConfusionMatrix(self, imgPredict, imgLabel):
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass ** 2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        return confusionMatrix


    def pixelAccuracy(self):
        acc = np.diag(self.confusionMatrix).sum() / self.confusionMatrix.sum()
        return acc
    

    def IntersectionOverUnion(self):
        # Intersection = TP Union = TP + FP + FN
        # IoU = TP / (TP + FP + FN)
        intersection = np.diag(self.confusionMatrix)  # 取对角元素的值，返回列表
        union = np.sum(self.confusionMatrix, axis=1) + np.sum(self.confusionMatrix, axis=0) - np.diag(
            self.confusionMatrix)  # axis = 1表示混淆矩阵行的值，返回列表； axis = 0表示取混淆矩阵列的值，返回列表
        IoU = intersection / union  # 返回列表，其值为各个类别的IoU
        return IoU
# 测试内容
if __name__ == '__main__':
    imgPredict = cv2.imread("../result/qqq.png")
    imgLabel = cv2.imread("../result/img.jpg")

    imgPredict = np.array(cv2.cvtColor(imgPredict, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    imgLabel = np.array(cv2.cvtColor(imgLabel, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)

    metric = SegmentationMetric(2)  # 2表示有2个分类
    ConfusionMatrix = metric.addBatch(imgPredict, imgLabel)


    IoU = metric.IntersectionOverUnion()

    print('IntersectionOverUnion is :\n', IoU)

效果图如下：
在这里插入图片描述

六、平均交并比MIoU（Mean Intersection Over Union）

将不同类别的IoU求个平均数

代码实现：

import numpy as np
import cv2

class SegmentationMetric(object):
    def __init__(self, numClass):
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,) * 2)  # 混淆矩阵（空）


    def addBatch(self, imgPredict, imgLabel):
        assert imgPredict.shape == imgLabel.shape
        self.confusionMatrix += self.genConfusionMatrix(imgPredict, imgLabel)  # 得到混淆矩阵
        return self.confusionMatrix


    def genConfusionMatrix(self, imgPredict, imgLabel):
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass ** 2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        return confusionMatrix


    def pixelAccuracy(self):
        acc = np.diag(self.confusionMatrix).sum() / self.confusionMatrix.sum()
        return acc


    def IntersectionOverUnion(self):
        # Intersection = TP Union = TP + FP + FN
        # IoU = TP / (TP + FP + FN)
        intersection = np.diag(self.confusionMatrix)  # 取对角元素的值，返回列表
        union = np.sum(self.confusionMatrix, axis=1) + np.sum(self.confusionMatrix, axis=0) - np.diag(
            self.confusionMatrix)  # axis = 1表示混淆矩阵行的值，返回列表； axis = 0表示取混淆矩阵列的值，返回列表
        IoU = intersection / union  # 返回列表，其值为各个类别的IoU
        return IoU


    def meanIntersectionOverUnion(self):
        mIoU = np.nanmean(self.IntersectionOverUnion())  # 求各类别IoU的平均
        return mIoU
# 测试内容
if __name__ == '__main__':
    imgPredict = cv2.imread("../result/qqq.png")
    imgLabel = cv2.imread("../result/img.jpg")

    imgPredict = np.array(cv2.cvtColor(imgPredict, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    imgLabel = np.array(cv2.cvtColor(imgLabel, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)

    metric = SegmentationMetric(2)  # 2表示有2个分类
    ConfusionMatrix = metric.addBatch(imgPredict, imgLabel)

    mIoU = metric.meanIntersectionOverUnion()

    print('meanIntersectionOverUnion is :\n', mIoU)

运行结果如下：
在这里插入图片描述

七、完整代码

import numpy as np
import cv2

__all__ = ['SegmentationMetric']

"""
confusionMetric  # 注意：此处横着代表预测值，竖着代表真实值
P\L     P    N
P      TP    FP
N      FN    TN
"""


class SegmentationMetric(object):
    def __init__(self, numClass):
        self.numClass = numClass
        self.confusionMatrix = np.zeros((self.numClass,) * 2)  # 混淆矩阵（空）

    def pixelAccuracy(self):
        # return all class overall pixel accuracy 正确的像素占总像素的比例
        #  PA = acc = (TP + TN) / (TP + TN + FP + TN)
        acc = np.diag(self.confusionMatrix).sum() / self.confusionMatrix.sum()
        return acc

    def classPixelAccuracy(self):
        # return each category pixel accuracy(A more accurate way to call it precision)
        # acc = (TP) / TP + FP
        classAcc = np.diag(self.confusionMatrix) / self.confusionMatrix.sum(axis=1)
        return classAcc  # 返回的是一个列表值，如：[0.90, 0.80, 0.96]，表示类别1 2 3各类别的预测准确率

    def meanPixelAccuracy(self):
        """
        Mean Pixel Accuracy(MPA，均像素精度)：是PA的一种简单提升，计算每个类内被正确分类像素数的比例，之后求所有类的平均。
        :return:
        """
        classAcc = self.classPixelAccuracy()
        meanAcc = np.nanmean(classAcc)  # np.nanmean 求平均值，nan表示遇到Nan类型，其值取为0
        return meanAcc  # 返回单个值，如：np.nanmean([0.90, 0.80, 0.96, nan, nan]) = (0.90 + 0.80 + 0.96） / 3 =  0.89

    def IntersectionOverUnion(self):
        # Intersection = TP Union = TP + FP + FN
        # IoU = TP / (TP + FP + FN)
        intersection = np.diag(self.confusionMatrix)  # 取对角元素的值，返回列表
        union = np.sum(self.confusionMatrix, axis=1) + np.sum(self.confusionMatrix, axis=0) - np.diag(
            self.confusionMatrix)  # axis = 1表示混淆矩阵行的值，返回列表； axis = 0表示取混淆矩阵列的值，返回列表
        IoU = intersection / union  # 返回列表，其值为各个类别的IoU
        return IoU

    def meanIntersectionOverUnion(self):
        mIoU = np.nanmean(self.IntersectionOverUnion())  # 求各类别IoU的平均
        return mIoU

    def genConfusionMatrix(self, imgPredict, imgLabel):  #
        """
        同FCN中score.py的fast_hist()函数,计算混淆矩阵
        :param imgPredict:
        :param imgLabel:
        :return: 混淆矩阵
        """
        # remove classes from unlabeled pixels in gt image and predict
        mask = (imgLabel >= 0) & (imgLabel < self.numClass)
        label = self.numClass * imgLabel[mask] + imgPredict[mask]
        count = np.bincount(label, minlength=self.numClass ** 2)
        confusionMatrix = count.reshape(self.numClass, self.numClass)
        # print(confusionMatrix)
        return confusionMatrix

    def Frequency_Weighted_Intersection_over_Union(self):
        """
        FWIoU，频权交并比:为MIoU的一种提升，这种方法根据每个类出现的频率为其设置权重。
        FWIOU =     [(TP+FN)/(TP+FP+TN+FN)] *[TP / (TP + FP + FN)]
        """
        freq = np.sum(self.confusion_matrix, axis=1) / np.sum(self.confusion_matrix)
        iu = np.diag(self.confusion_matrix) / (
                np.sum(self.confusion_matrix, axis=1) + np.sum(self.confusion_matrix, axis=0) -
                np.diag(self.confusion_matrix))
        FWIoU = (freq[freq > 0] * iu[freq > 0]).sum()
        return FWIoU

    def addBatch(self, imgPredict, imgLabel):
        assert imgPredict.shape == imgLabel.shape
        self.confusionMatrix += self.genConfusionMatrix(imgPredict, imgLabel)  # 得到混淆矩阵
        return self.confusionMatrix

    def reset(self):
        self.confusionMatrix = np.zeros((self.numClass, self.numClass))


# 测试内容
if __name__ == '__main__':
    imgPredict = cv2.imread("../result/qqq.png")
    imgLabel = cv2.imread("../result/img.jpg")
    #"../result/standard/mask/SM50GRADE1PLAIN(1).jpg"
    #"../result/predict/image/SM50GRADE1BLANKET(1).jpg"

    imgPredict = np.array(cv2.cvtColor(imgPredict, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    imgLabel = np.array(cv2.cvtColor(imgLabel, cv2.COLOR_BGR2GRAY) / 255., dtype=np.uint8)
    # imgPredict = np.array([0, 0, 1, 1, 2, 2])  # 可直接换成预测图片
    # imgLabel = np.array([0, 0, 1, 1, 2, 2])  # 可直接换成标注图片

    metric = SegmentationMetric(2)  # 2表示有2个分类，有几个分类就填几
    ConfusionMatrix = metric.addBatch(imgPredict, imgLabel)
    pa = metric.pixelAccuracy()
    cpa = metric.classPixelAccuracy()
    mpa = metric.meanPixelAccuracy()
    IoU = metric.IntersectionOverUnion()
    mIoU = metric.meanIntersectionOverUnion()
    print('ConfusionMatrix is :\n', ConfusionMatrix)
    print('PA is : %f' % pa)
    print('cPA is :', cpa)
    print('mPA is : %f' % mpa)
    print('IoU is : ', IoU)
    print('mIoU is : ', mIoU)

效果图如下：
在这里插入图片描述

猜你喜欢

转载自blog.csdn.net/qq_41264055/article/details/128000668
今日推荐
《美国对全球网络空间安全与发展的威胁和破坏》报告发布
火速冲上 GitHub 热榜 —— 开源编程语言、框架哪有这么可爱？
北京人形机器人创新中心发布全球首个纯电驱拟人奔跑的全尺寸人形机器人“天工”
LFOSSA 源来如此公开课 | 掌握云原生未来：CNCF 认证全面攻略与备考秘籍
周排行
循环神经网络（rnn）讲解
Tigao教程四：单独的关节运动
金蝶K3WISE15.0-注册套打教程
如何在Mac上配置Kubernetes
Android应用结束自身进程的方法
SpringMVC学习 十三 拦截器栈
中国驻洛杉矶总领馆举行新春招待会
HttpClient get post 发送
11 - three.js 笔记 - 绘制三维字体模型
Mysql递归获取某个父节点下面的所有子节点和子节点上的所有父节点
每日归档
更多
2024-05-01(4)
2024-04-30(1)
2024-04-29(40)
2024-04-28(0)
2024-04-27(56)
2024-04-26(39)
2024-04-25(22)
2024-04-24(36)
2024-04-23(26)
2024-04-22(39)

代码天地 © 2018 codetd.com

境外服务器   最新电视剧2022