Hinge Loss 和 Zero-One Loss

Hinge Loss 和 Zero-One Loss

Wikipedia: https://en.wikipedia.org/wiki/Hinge_loss

Chart description:

• The vertical axis represents fixed $t=The\; values\; of\; Hinge\; loss\; (blue)\; and\; Zero-One\; Loss\; (green)\; of\; 1$ , while the horizontal axis represents the predicted value$the\; value\; of\; y$ .
• The figure shows that Hinge loss penalizes the predicted value $y<1$ , which corresponds to the concept of margin in support vector machines.

Hinge Loss

Hinge Loss is a commonly used machine learning loss function, usually used for classification problems in support vector machine (SVM) models. The function is defined as follows:
$$\begin{gathered} L(y,f(x))=\max (0,1-y_i\cdot f(x)) \\ f(x)=w^{\mathrm{T}}{x}_i+b\text{\hspace{1em}( 1 )} \end{gathered}$$
Among them,$y_i$is the true label of the sample, $f\left(x\right)$ is the predicted value of the model. The value range of this function is a non-negative real number. When the error between the predicted value and the real value is larger, the value of the loss function is larger.

When the sample is correctly classified, that is, $y_i\cdot f(x)>0$ , at this time, the value of Hinge Loss is 0, indicating that the model classification is correct and no error occurs.

When the sample is misclassified, that is, $y_i\cdot f\left(x\right)<0$ , at this time the value of Hinge Loss is$1-y_i\cdot f\left(x\right)$ indicates the classification error of the model, and the greater the classification error, the greater the value of Hinge Loss.

The goal of Hinge Loss is to minimize the classification error while encouraging the model to produce large margins (i.e. the distance between the correct classification and the classification hyperplane).

In support vector machines, the goal is to find a hyperplane with the largest interval to classify samples, so Hinge Loss can be associated with the interval. For a sample point $(x_i,y_i)$ , the distance between it and the hyperplane is:
$$\frac{y_i{w}^T{x}_i+b}{\Vert w\Vert }\text{\hspace{1em}( 2 )}$$
Among them,$w$ and$b$ are the weights and biases in the SVM model. Record this distance as$c_i$, Hinge Loss can be re-expressed as:
$$L(y_i,f(x_i))=\max (0,1-y_i(c_i\Vert w\Vert ))\text{\hspace{1em}( 3 )}$$
Therefore, Hinge Loss can not only express the classification error, but also promote the model to produce a larger interval, thereby increasing the generalization ability of the model.

Zero-One Loss

Zero-One Loss is a common classification loss function in machine learning. For a binary classification problem, suppose $y\in -1,1$ is the real label,$f\left(x\right)$ is the model pair samplexxInstead of$x$
$$L(y,f(x))=\{\begin{array}{ll}0& \text{if }y=f(x)\\ 1& \text{otherwise}\end{array}\text{\hspace{1em}( 4 )}$$
That is to say, when the prediction result of the model is consistent with the real label, Zero-One Loss is 0; otherwise, Loss is 1. It can be seen from the expression that Zero-One Loss has a very high penalty for prediction errors, because no matter how close the wrong prediction is to the correct one, Loss will be calculated as 1. Compared with other loss functions, Zero-One Loss is often considered a very strict evaluation method.

However, since Zero-One Loss itself is not derivable, it is usually chosen to use some derivable approximation functions, such as Hinge Loss or Cross Entropy Loss, etc. when training the model. Compared with Zero-One Loss, these loss functions are smoother, which can help the model converge faster and more stably.

It should be noted that although Zero-One Loss is very strict in evaluating model performance, it is often not the optimal choice in practical applications. Especially when there is some noise in the labels in the data set, using Zero-One Loss may cause the model to overfit the training set and fail to effectively generalize to the test set. Therefore, in practical applications, we usually use a smoother loss function, combined with some common regularization techniques, such as L1/L2 regularization, to control the complexity and generalization ability of the model.