Confusion Matrix

A confusion matrix is a statistical result based on the prediction results of a classification model at a certain threshold.

Binary classification task

Each element of the confusion matrix represents the number of results for the model to classify samples of different classes under a certain threshold . The composition of the confusion matrix is as follows:

                预测为负例       预测为正例
实际为负例     True Negative (TN)   False Positive (FP)
实际为正例     False Negative (FN)   True Positive (TP)

Among them, TN represents the number of true negative cases, FP represents the number of false positive cases, FN represents the number of false negative cases, and TP represents the number of true cases.

TP (True Positive) indicates that the model correctly predicts the number of positive examples (Positive) as positive examples, the real value is positive, and the model considers the number of positive examples (True Positive=TP)

FN (False Negative) indicates the number of positive examples that the model predicts as negative examples (Negative), the true value is positive, and the number that the model considers negative (False Negative=FN): This is the second type of error in statistics (Type II Error)

FP (False Positive) indicates the number of negative examples that the model predicts as positive examples, the true value is negative, and the number that the model considers positive (False Positive=FP): This is the first type of error in statistics (Type I Error)

TN (True Negative) indicates that the model correctly predicts the number of counterexamples as counterexamples, the true value is negative, and the number that the model considers negative (True Negative=TN)

Based on the confusion matrix, we can calculate a variety of evaluation indicators to measure the performance of the model , such as accuracy (Accuracy), precision (Precision), recall (Recall) and F1 score (F1 Score), etc.

Accuracy

Precision

Recall rate (Recall)

Calculation formula of F1 Score

multi-category task

Confusion Matrix is a common tool for evaluating the performance of classification models. The confusion matrix shows the statistical information of the classification results of the model for multiple categories of samples.

For a K-category multi-classification task, the size of the confusion matrix is K x K, where each row represents the true class and each column represents the predicted class. Each element (i, j) of the matrix represents the number of samples of true class i predicted by the model as class j.

For this confusion matrix, the abscissa is the ground truth, and the ordinate is the category predicted by the model. The diagonal line is the information we are most concerned about, and the diagonal line represents the number of samples that are predicted correctly.

For the calculation of multi-classification accuracy (Accuracy), precision (Precision), recall (Recall) and F1 score (F1 Score), etc., multi-classification is often converted into binary classification for calculation.

ROC curves for binary and multiclass classification

Receiver operating characteristic (receiveroperatingcharacteristic) , each point on the ROC curve reflects the sensitivity to the same signal stimulus.

ROC curve for two categories:

In the binary classification task, the ROC curve is drawn with the True Positive Rate (TPR) as the vertical axis and the False Positive Rate (FPR) as the horizontal axis.

The ROC curve shows the performance of the model under different thresholds.

Usually, the ROC curve in the binary classification task is obtained by calculating the TPR and FPR of the model at different thresholds. According to the predicted probability or score of the model, different thresholds can be selected, the corresponding TPR and FPR can be calculated, and the ROC curve can be drawn. The area under the curve (Area Under the Curve, AUC) is a common indicator to measure the performance of the ROC curve. The larger the AUC value, the better the performance of the model.

Multi-category ROC curve:

The ROC curve in the multi-classification task is also drawn with TPR and FPR as coordinates, but in the case of multi-classification, the TPR and FPR.

Micro-average: The number of true positive cases and false positive cases of all categories is accumulated, and then the overall TPR and FPR are calculated. A micro-averaged ROC curve can then be plotted.
Macro average: Calculate the TPR and FPR of each category separately, and then calculate the average to get the macro average TPR and FPR. The macro-averaged ROC curve can then be plotted.

The ROC curve in the multi-classification task can be drawn using a method similar to that in the binary classification task, but it is necessary to calculate the TPR and FPR of multiple categories according to the specific calculation method, and consider using micro-average or macro-average to draw the corresponding ROC curve.

Matplotlib interactive chart - confusion matrix visualization

Python realizes the ROC curve of binary classification and multi-classification

ROC curve and AUC value of machine learning classifier performance indicators

ML---Python draws confusion matrix, PR curve, ROC curve