Confusion matrix (Confusion Matrix)
In the field of machine learning, confusion matrix (Confusion Matrix), also known as the likelihood of a matrix or error matrix. The confusion matrix is a visualization tool, especially for supervised learning, unsupervised learning in general is called the matching matrix. In the image evaluation precision, mainly for comparison and classification results actually measured value, the accuracy of the classification results are displayed in a confusion matrix inside.
Confusion matrix configuration diagram showing the general method is as follows.
Confusion matrix to express meanings:
- Each column represents the prediction class confusion matrix, each column represents the number of the total number of prediction for the category;
- Each row represents a real home category data, the total number of each row of data represents the number of data instances of this class; the value in each column represents the number of real data is predicted class.
True Positive (TP): real class. Real class sample is positive type, and model identification result is positive class.
False Negative (FN): false negative category. Category samples are positive real class, but to recognize it as a negative model class.
False Positive (FP): false positive class. Real class sample is negative class, but to recognize it as a positive model class.
True Negative (TN): negative real class. True category class sample is negative, and recognize it as a negative model class.
The second-class classification matrix can be used readily appreciated, but by adding more rows and columns of the confusion matrix can be easily applied to a three or more types of questions values.
For example
If the sample data 150, predicted to Class I, II, III 50 each. After the end of the classification confusion matrix obtained as follows:
Each row represents the sum of the true number of samples that category, and each column represents the sum of the number of samples to be predicted for the category.
The first row of the first column 43 represents 43 belongs to the first instance of the actual type is predicted to be a first type, the same token, the first row of the second column 2 are two practical examples of the first category of home the second kind is mispredicted.
The confusion matrix is a summary of predictions for classification problems. Use the correct summary count value and the number of incorrect predictions, according to a breakdown of each class, which is the key to the confusion matrix. Confusion matrix shows which part will cause confusion when a classification model making a prediction. It not only allows you to understand the mistakes of the classification model, is more important is to understand what type of error is occurring. It is this decomposition of the results overcome the limitations of using only classification accuracy brings.
Obtained from the confusion matrix classification index
From among the confusion matrix, you can get higher classification index: Accuracy (accuracy rate), Precision (accuracy rate or accuracy), Recall (recall), Specificity (specific), Sensitivity (sensitivity).
Calculation
For binary classification, the sample can be divided into categories according to their true combination of machine learning is predicted category is as follows:
The total number of samples = TP + FP + TN + FN.
It is possible to calculate the index:
Accuracy rate (Accuracy): accuracy rate is the most commonly used classification performance. It can be used to indicate the accuracy of the model, i.e. the total number of models to identify the correct number / sample. In general, the higher the accuracy of the model, the better the model results.
Accuracy = (TP+TN)/(TP+FN+FP+TN)
Accuracy or accuracy (Precision): also known as precision, represents the model identified as positive sample class, the real proportion of samples positive for the class of share. In general, the higher the precision, the better the effect of the model.
Precision = TP/(TP+FP)
Recall (Recall): also known as recall , recall showed the actual positive samples, the classifier can predict how much out.
Recall (recall) = Sensitivity (Sensitive Index, True Positive Rate, TPR) = recall
It represents the model of the total number of positive samples of the positive class class ratio of the number of samples correctly identified. Generally, Recall higher, indicating a more positive samples were correctly predicted the class, the better the model model.
Recall = TP/(TP+FN)
Precision and recall rate is an indicator of a contradiction. In general, when the precision rate and the recall rate Want low; Second recall rate, precision often low.
The difference between accuracy rate (Accuracy) and accuracy (Precision) of
Accuracy, no matter what category, as long as the forecast is correct, that are placed on the number of molecules, and the denominator is the total amount of data to prove that the accuracy rate is the judgment of all the data.
The accuracy in the corresponding category is a category, the category is correct molecular number forecast, the denominator is the number predicted for all categories.
Or, Accuracy is the evaluation of the overall accuracy of the classification, and Precision is a classifier to predict the precise evaluation of a particular category.
Specificity (specific): specific marker, indicates the number of samples of the model identification of negative type, the total number of samples of the ratio of negative type.
Negative positive class rate (False Positive Rate, FPR), calculated as follows: FPR = FP / (TN + FP), calculated by the model fault is identified as a negative type samples positive class proportion of all negative class samples, generally lower the better.
Specificity = 1 - FPR
F. Β_Score: physical meaning is to Fβ Weighted average precision and recall rate, in the process of merging, the recall of a weight of β times the correct rate.
F1 score believes recall and accuracy are equally important, F2 fraction considered the importance of the recall is correct ratio of 2 times, while F0.5 score believes the importance of the recall rate is half correct rate. F1 is the more common fraction (F1 Score), is used to measure an indicator of the statistical accuracy of the two classification model.
F1_Score: mathematical definition for the F1 fraction (F1-Score), also known as the balance of F scores (Balanced Score) , which is defined as the harmonic mean of precision and recall rates. In the case of β = 1, the value of F1-Score is from 0 to 1, 1 is the best, 0 the worst.
So we know that computing Precision, Recall, Specificity and other computing features just a classification and Accuracy and F1-Score model is to determine the overall classification criteria.
About AUC
http://fastml.com/what-you-wanted-to-know-about-auc/
AUC, or Area Under Curve, is a metric for binary classification. It’s probably the second most popular one, after accuracy. Unfortunately, it’s nowhere near as intuitive. That is, until you have read this article.
Accuracy deals with ones and zeros, meaning you either got the class label right or you didn’t. But many classifiers are able to quantify their uncertainty about the answer by outputting a probability value. To compute accuracy from probabilities you need a threshold to decide when zero turns into one. The most natural threshold is of course 0.5.
Let’s suppose you have a quirky classifier. It is able to get all the answers right, but it outputs 0.7 for negative examples and 0.9 for positive examples. Clearly, a threshold of 0.5 won’t get you far here. But 0.8 would be just perfect.
That’s the whole point of using AUC - it considers all possible thresholds. Various thresholds result in different true positive/false positive rates. As you decrease the threshold, you get more true positives, but also more false positives. The relation between them can be plotted:
Image credit: Wikipedia
From a random classifier you can expect as many true positives as false positives. That’s the dashed line on the plot. AUC score for the case is 0.5. A score for a perfect classifier would be 1. Most often you get something in between.
Here is some of the information found on the web
ROC / AUC concept
1. The sensitivity, specificity, true rate, false positive rate
Before the formal presentation ROC / AUC, we still have to introduce two indicators, choose these two indicators is also the ROC and AUC can ignore the causes of the imbalance of the sample. These two indicators are: sensitivity and (1-specificity), also known as the real rate (TPR) and false positive rate (the FPR) .
Sensitivity (Sensitivity) = TP / (TP + FN)
Specificity (Specificity) = the TN / (the FP + the TN)
- In fact, we can see that the sensitivity and the recall rate is exactly the same, only the names changed it.
- As we are more concerned about the positive sample, it is necessary to see how many negative samples were incorrectly predicted to be positive samples, so the use of (1-specificity), rather than specificity.
Real rate (TPR) = Sensitivity = TP / (TP + FN)
False Positive Rate (FPR) = 1- specificity = the FP / (the FP + the TN)
Below is a schematic true and false-positive rate, we found TPR and FPR are based on the actual performance of 1s and 0s departure, that they were in a real positive and negative samples in the probability to observe the relevant issues. Because of this, so no matter whether the sample balance, it will not be affected. Or take the previous example, the total sample, 90% of the sample is positive, 10% negative samples. We know with accuracy is water, but with TPR and FPR is not the same. Here, TPR focus only 90% of the positive samples how much is actually covered, and that 10% nothing to do with, empathy, FPR only concerned 10% of the negative samples how much is wrong coverage, and also that 90 % There is no relationship, so we can see: if we start from the perspective of the actual performance of each result, you can avoid the problem of imbalance in the sample, which is why the choice of TPR and FPR as the reason for ROC / AUC indicators.
Learning to see the article raised a question regarding: specificity (Specificity) and sensitivity (Sensitivity)
Record, there is no understanding.
Consider a case where two categories, category 0 and 1, 1 and 0, respectively, will be a positive type (positive) and negative type (negative), then the actual results of the classification has four, the following table:
From this table can be drawn out some other evaluation:
- The ACC : Classification Accuracy, described classification accuracy of the classifier
is calculated as follows: ACC = (TP + TN) / (TP + FP + FN + TN) - The BER : error Balanced Rate
is calculated as: BER = 1/2 * ( FPR + FN / (FN + TP)) - TPR : Rate to true positive, n-described embodiment all the identified proportion of all the positive examples
is calculated as: TPR = TP / (TP + FN) - The FPR : Rate to false positive, negative embodiment described the case of positive identification as the proportion of all cases of negative examples
is calculated as follows: FPR = FP / (FP + TN) - The TNR : Rate to true negative, negative examples describe the identified proportion of all the negative examples
is calculated as: TNR = TN / (FP + TN) - The PPV : Positive Predictive value
is calculated as: PPV = TP / (TP + FP) - The NPV : Negative Predictive value
calculation formula: NPV = TN / (FN + TN)
wherein the TPR is the sensitivity (Sensitivity), the TNR is the specificity (specificity).
Wikipedia drawings:
The following is a screenshot taken on wiki (https://en.wikipedia.org/wiki/Confusion_matrix)
