1. Accuracy The
accuracy_score function calculates the classification accuracy: returns the proportion or number of samples that are correctly classified.
When in a multi-label classification task, this function returns the accuracy of the subset. For a given sample, if the predicted label set is The true label set of the sample is consistent, then subset accuracy=1, otherwise zero
import numpy as np
from sklearn.metrics import accuracy_score
y_pred=[0,2,1,3]
y_true=[0,1,2,3]
print(accuracy_score(y_true,y_pred,normalize=False))
2
print(accuracy_score(y_true,y_pred))
0.52. Confusion matrix or table of confusion
TP (True Positive): predict the positive class as a positive class number, the true value is 0, and the prediction is also 0
FN (False Negative): predict the positive class as a negative class number, the true value is 0, and the prediction is 1
FP (False Positive) : Predict the negative class as a positive class number, the true value is 1, the prediction is 0
TN (True Negative): the negative class is predicted as a negative class number, the true value is 1, and the prediction is also 1
from sklearn.metrics import confusion_matrix
y_true=[2,0,2,2,0,1]
y_pred=[0,0,2,2,0,2]
print(confusion_matrix(y_true,y_prednlabels=[]))3. The
commonly used commands of precious-recall-Fmeasur are as follows.
Through the confusion matrix (as shown in the figure below)
, the formula of each parameter can be clearly seen
. The name of the common measurement parameter and the solution method.
Accuracy means that the test result and the measurement point are very
accurate. Means that the test result is close to the true value.
Parameter F=(1+β²) pre recall/(β²prec+recall) The
smaller the β, the greater the weight of prec, and vice versa, the greater the recall weight. When β==1, both are equally important.
from sklearn import metrics
y_pred = [0, 1, 0, 1, 0, 0]
y_true = [1, 0, 0, 1, 0, 0]
print(metrics.precision_score(y_true, y_pred))
#precision_score仅支持二元分类，及0，1分类
print(metrics.recall_score(y_true, y_pred))
print(metrics.fbeta_score(y_true, y_pred,beta=1))
#f参数需要指定β的大小
print(metrics.precision_recall_fscore_support(y_true, y_pred,beta=1))import numpy as np
from sklearn.metrics import precision_recall_curve
from sklearn.metrics import average_precision_score
y_true = np.array([0, 0, 1, 1])
y_score = np.array((0.1, 0.4, 0.35, 0.8))
prec, recall, threshold = precision_recall_curve(y_true, y_score)
print(prec)
print(recall)
print(threshold)
print(precision_recall_curve(y_true, y_score))In order to observe these three indicators in ython, they are encapsulated in a function, and classification_report
from sklearn.metrics import classification_report
y_pred = [0, 1, 0, 1, 0, 0]
y_true = [1, 1, 0, 1, 0, 0]
target = ['class1', 'class2']
print(classification_report(y_true, y_pred, target_names=target))
#上述代码中class的个数取决于数据集的维数，二元分类就只有两个class，三元就3个，无法多设置或少设置，说明report可以操作多元分类How to extend binary classification to multi-class or multi-label problems?
By default, only positive labels are used to calculate indicators. In order to extend these indicators to multiple categories, we treat multiple categories as a set of binary classifications and divide the data set. Calculate the binary indicators of these sub-categories at the same time, and average the scores on all sub-category problems, usually using average.
5 ways to handle weights: macro, weighted, mirco, samples, average
4.
ROC curve ROC space (also called sensitivityVS1-sensitivity plot) uses TPR (also called sensitivity) as the T axis and
FPR (1-sensitivity) as the X axis to form a two-dimensional coordinate space. ROC space describes TP and The principle of a trade-off between FP. The point set on the upper left indicates that the situation is better, and the lower right indicates that the situation is worse.
In a binary classification problem, the category prediction of each sample instance is usually made based on a continuous random variable X, which is calculated from the sample instance The random variable X is called score. Given a threshold parameter T, X>T means that the positive class obeys F1(X), and X<T is called the negative class and obeys F2(X) distribution. Therefore, TPR and FPR are repeated integrations of value intervals. At this time, TPR and FPR can be regarded as a function of the integral limit t, forming a point set of (TPR(t), FPR(t)). By changing the range of t, the ROC curve can be obtained
sklearn.metrics.roc_curve(y_ture,y_score,pos_label=None,
sample_weight=None,drop_intermediate=True)import numpy as np
from sklearn.metrics import roc_curve
from sklearn.metrics import roc_auc_score
y = np.array([1, 1, 2, 2])
scores = np.array([0.1, 0.4, 0.35, 0.8])
fpr, tpr, thresholds = roc_curve(y, scores, pos_label=2)
print(fpr)
print(tpr)
print(thresholds)
print(roc_auc_score(y,scores))#计算AUC值5. Loss function
For a given input X, the learner model predicts the corresponding result. The gap between the predicted response and the true response is described by the loss function
Common losses are:
zero_one loss
from sklearn.metrics import zero_one_loss
y_pred = [1,2,3,4]
y_true = [2,2,3,4]
print(zero_one_loss(y_true, y_pred))square loss (not commonly
used in classifiers) hing loss (commonly used in SVM)
log loss or cross-entropy loss The
log loss function judges whether the probability distribution of the predicted output matches the true distribution, rather than judging whether the label classes are equal.
For class label Y, 0 and 1 can be taken, and the loss size of different class labels is different from the probability P.
For multiple classification, there is also a corresponding loss function
from sklearn.metrics import log_loss
y_true = [0, 0, 1, 1]
y_pred = [(0.4, 0.6), (0.2, 0.8), (0.5, 0.5), (0.2, 0.8)]#调用log损失的时候，会默认生成one-hot矩阵，如上图，如果两个标签，那么只能有两个分布概率
print(log_loss(y_true, y_pred))
y_true = [0, 1, 2, 4]
y_pred = [(0.4, 0.2, 0.2, 0.2), (0.2, 0.4, 0.2, 0.2), (0.5, 0.1, 0.1, 0.3), (0.3, 0.3, 0.3, 0.1)]
print(log_loss(y_true, y_pred))
1.6094379124341
logistic loss
hamming_loss
represents the Hamming distance of the data set
from sklearn.metrics import hamming_loss
import numpy as np
y_true = [1, 2, 2, 5]
y_pred = np.array([1, 2, 3, 5])
print(hamming_loss(y_true, y_pred))#只有在3处一个为2一个3，所以是1/4=0.25
0.25