[Machine Learning] Reliability Curve and Calibration of Probabilistic Model

1. What is a reliability curve?

The reliability curve is an evaluation index for a probabilistic model and is suitable for probabilistic algorithms such as Naive Bayes, SVM, and logistic regression. It is a curve with the predicted value of y as the abscissa and the actual value of y as the ordinate.
Therefore, when the reliability curve we draw is closer to the diagonal, we think the performance of this learner is better.

2. Code display

The reliability curve of sklearn is the same as the learning curve. It cannot draw the graph directly but returns the values ​​required for drawing. Therefore, the following code explores the reliability curves of the three algorithms based on the values ​​returned by calibration_curve.

from sklearn.datasets import make_classification as mc
import matplotlib.pyplot as plt
from sklearn.naive_bayes import GaussianNB # 导入高斯朴素贝叶斯
from sklearn.svm import SVC
from sklearn.linear_model import LogisticRegression as LR
from sklearn.metrics import brier_score_loss # 导入布里尔分数
from sklearn.model_selection import train_test_split
from sklearn.calibration import calibration_curve # 对概率类模型进行校准，方法是分箱


#创建数据
x,y = mc(n_samples=100000,
         n_features= 20,
         n_classes= 2