1. Description
In machine learning, performance measurement is a fundamental task. Therefore, when it comes to classification problems, we can rely on the AUC - ROC curve. When we need to examine or visualize the performance of a multi-class classification problem, we use AUC (Area Under the Curve ) ROC ( Receiver Operating Characteristic ) curves. It is one of the most important evaluation metrics to check the performance of any classification model. It is also written as AUROC ( Area Under Receiver Operating Characteristics )
Note: For a better understanding, I recommend you to read my article on confusion matrices .
This blog aims to answer the following questions:
1. What is AUC - ROC curve?
2. Define the terms used in the AUC and ROC curves.
3. How to infer the performance of the model?
4. Relationship between sensitivity, specificity, FPR and threshold.
5. How to use AUC - ROC curve in multiclass model?
2. What is the AUC-ROC curve?
AUC - ROC curve is a performance measure for classification problems at various threshold settings. ROC is a probability curve and AUC represents the degree or measure of separability. It tells the degree to which the model is able to distinguish classes. The higher the AUC, the better the model is at predicting 0 classes as 0 and 1 classes as 1. By analogy, the higher the AUC, the better the model is at distinguishing between patients with and without the disease.
The ROC curve is plotted using TPR versus FPR, where TPR is on the y-axis and FPR is on the x-axis.
3. Define the terms used in AUC and ROC curves.
3.1 TPR (True Positive Rate)/Recall/Sensitivity
3.2 Specificity
3.3 FPR
4. How to infer the performance of the model?
A good model has an AUC close to 1, which means it has good separability. The poorer model has an AUC close to 0, which means it has the worst separability measure. In fact, it means that it is returning the result. It predicts 0 as 1 and 1 as 0. When the AUC is 0.5, it means that the model does not have any class separation ability. Let's interpret the above statement. As we all know, ROC is a probability curve. So let's plot the distribution of these probabilities: Note: The red distribution curve is the positive class (patients with the disease), and the green distribution curve is the negative class (patients without the disease).
This is an ideal situation. When two curves do not overlap at all, it means that the model has an ideal separability measure. It is perfectly capable of distinguishing between positive and negative classes.
When two distributions overlap, we introduce Type 1 and Type 2 errors. Depending on the threshold, we can minimize or maximize them. When the AUC is 0.7, it means that the model has a 70% chance of being able to distinguish between positive and negative classes.
This is the worst case scenario. When the AUC is about 0.5, the model has no discriminative ability to distinguish between positive and negative classes.
When the AUC is around 0, the model is actually reciprocating the classes. This means that the model predicts negative classes as positive and vice versa.
5. Relationship between sensitivity, specificity, FPR and threshold.
Sensitivity and specificity are inversely proportional. So when we increase the sensitivity, the specificity decreases and vice versa.
Sensitivity, specificity and sensitivity ⬆️⬇️, specificity ⬇️⬆️
When we lower the threshold, we get more positive values, increasing sensitivity and decreasing specificity.
Also, when we increase the threshold, we get more negative values, resulting in higher specificity and lower sensitivity.
It is well known that FPR is 1-specific. So when we increase TPR, FPR also increases and vice versa.
TPR，FPR和TPR，FPR⬆️⬆️⬇️⬇️
6. How to use AUC ROC curve in multi-class model?
In a multi-class model, we can use the One vs ALL method to plot N AUC ROC curves for N classes. For example, if you have three classes named X, Y, and Z , you will have one ROC for classifying X on Y and Z , another ROC for classifying Y on class Y, and a third for classifying Y and X Z.