Loss function (Loss)

If we define a machine learning model, such as a three-layer neural network, then we need to make this model fit the provided training data as closely as possible. But how do we evaluate whether the model fits the data well enough? Then you need to use corresponding indicators to evaluate its fitting degree. The function used is called the loss function . When the value of the loss function decreases, we think that the model has taken another step forward on the road of fitting. The best fit of the final model to the training data set is when the value of the loss function is minimum, which is when the average value of the loss function is minimum on the specified data set.

Cross Entropy loss function (Cross Entropy)

In physics, "entropy" is used to describe the degree of disorder exhibited by a thermodynamic system. Shannon introduced this concept into the field of information theory and proposed the concept of "information entropy" to measure the uncertainty of information through a logarithmic function.

Cross entropy is an important concept in information theory, which is mainly used to measure the difference between two probability distributions. Assume $p$ and$q$ is dataxxTwo probability distributions of$x$$q$ to represent$The\; cross\; entropy\; of\; p$ can be calculated as follows:$$H(p,q)=-\sum _{x}p(x)\log q(x))$$ cross entropy characterizes the distance between two probability distributions, aiming to depict the probability distribution$q$ to express the probability distributionppThe degree of difficulty of$p\; .$It is not difficult to understand according to the formula. The smaller the cross entropy, the two probability distributions$p$ and$q$ is closer.

Here we still take the three-category classification problem as an example, assuming that the data $x$ belongs to category 1. Remember data$The\; class\; distribution\; probability\; of\; x$ is$y$，显然 $y=(1,0,0)$ represents dataxxThe actual class distribution probability of$x\; .$Remember$\hat{y}$Represents the class distribution probability predicted by the model.

So for data $For\; x$ , its actual class distribution probability$y$ and the model predicted class distribution probability$\hat{y}$The cross entropy loss function is defined as:
$$crossentropy=-y\times \log \left(\hat{y}\right)$$

Obviously, a good neural network should try to ensure that for each input data, the gap between the category distribution probability predicted by the neural network and the actual category distribution probability is as small as possible, that is, the smaller the cross entropy, the better. Therefore, cross entropy can be used as a loss function to train neural networks.

The figure above gives an example of three category classification. Since the input data $x$ belongs to category 1, so its actual category probability distribution value is$y=(y_1,y_2,y_3)=(1,0,0)$ . After transformation by the neural network, the input data xxis obtained$x$ relative to the predicted median value of the three categories$(z_1,z_2,z_3)$ . Then, through softmax function mapping, the input data xxpredicted by the neural network is obtainedClass distribution probability of$x$$\hat{y}=({\hat{y}}_1,{\hat{y}}_2,{\hat{y}}_3)$ . According to the previous introduction,${\hat{y}}_1,{\hat{y}}_2,{\hat{y}}_3$is a probability value between the range (0,1). Due to sample $x$ belongs to the first category, so we hope that the neural network predicts${\hat{y}}_1$The value must be much larger than ${\hat{y}}_2$和${\hat{y}}_3$value. In order to obtain such a neural network, the following cross-entropy loss function can be used to optimize the model parameters during training: $$crossentropy=-\left(y_1\times \log \left({\hat{y}}_1\right)+y_2\times \log \left({\hat{y}}_2\right)+y_3\times \log \left({\hat{y}}_3\right)\right)$$ In the above formula,$y_2$和$y_3$are all 0, $y_1$is 1, so the cross-entropy loss function is simplified to: $$-y_1\times \log \left({\hat{y}}_1\right)=-\log \left({\hat{y}}_1\right)$$ In neural network training, the error (i.e. loss) between the actual category probability distribution of the input data and the category probability distribution predicted by the model is transferred from the output end to the input end in order to optimize the model parameters. The following is a brief introduction to the error calculated based on cross entropy from${\hat{y}}_1$passed to $z_1$sum $z_2$（$z_3$The derivation of and $z_2$ 相同）的情况。$$\frac{\partial {\hat{y}}_1}{\partial z_1}=\frac{\partial \left(\frac{e^{z_1}}{{\sum }_k{e}^{z_k}}\right)}{\partial z_1}=\frac{{\left(e^{z_1}\right)}^{{}^{\prime }}\times {\sum }_k{e}^{z_k}-e^{z_1}\times e^{z_1}}{{\left({\sum }_k{e}^{z_k}\right)}^2}=\frac{e^{z_1}}{{\sum }_k{e}^{z_k}}-\frac{e^{z_1}}{{\sum }_k{e}^{z_k}}\times \frac{e^{z_1}}{{\sum }_k{e}^{z_k}}={\hat{y}}_1\left(1-{\hat{y}}_1\right)$$ due to the cross entropy loss function$-\log \left({\hat{y}}_1\right)$对${\hat{y}}_1$The result of the derivation is $-\frac{1}{{\hat{y}}_1}$，${\hat{y}}_1\left(1-{\hat{y}}_1\right)$与$-\frac{1}{{\hat{y}}_1}$The result of multiplication is ${\hat{y}}_1-1This$ shows that once the model prediction output${\hat{y}}_1$, subtracting 1 from this output is the cross loss function relative to $z_1$ 的偏导结果。$$\frac{\partial {\hat{y}}_1}{\partial z_2}=\frac{\partial \left(\frac{e^{z_1}}{{\sum }_k{e}^{z_k}}\right)}{\partial z_2}=\frac{0\times {\sum }_k{e}^{z_k}-e^{z_1}\times e^{z_2}}{{\left({\sum }_k{e}^{z_k}\right)}^2}=-\frac{e^{z_1}}{{\sum }_k{e}^{z_k}}\times \frac{e^{z_2}}{{\sum }_k{e}^{z_k}}=-{\hat{y}}_1{\hat{y}}_2$$In the same way, the derivative of the cross entropy loss function is $-\frac{1}{{\hat{y}}_1}$，$-{\hat{y}}_1{\hat{y}}_2$与$-\frac{1}{{\hat{y}}_1}$The result of multiplication is ${\hat{y}}_2$. This means that partial derivatives are performed for nodes other than the first output node. After getting the model prediction output, as long as it is saved, it is the partial derivative result of the cross loss function relative to other nodes. at $z_1$、$z_2$sum $z_3$After obtaining the partial derivative result, the loss error can be transferred to the input end through the chain rule (described later).

In the above example, assume that the predicted intermediate values $(z_1,z_2,z_3)$ after Softmax mapping, the result is$(0.34,0.46,0.20)$ . Since the input data$x$ belongs to the first category. Obviously this output is not ideal and the model parameters need to be optimized. If the cross-entropy loss function is chosen to optimize the model, then$(z_1,z_2,z_3)$ The partial derivative value of this layer is$(0.34-1,0.46,0.20)=(-0.66,0.46,0.20)$。

It can be seen that the combination of Softmax and cross-entropy loss functions brings great convenience to partial derivative calculations. The partial derivative calculation causes the loss error to be transferred from the output end to the input end to optimize the model parameters. Here, the cross-entropy is combined with the Softmax function, so it is also called Softmax loss (Softmax with cross-entropy loss).

Mean Square Error (MSE)

Mean square error loss is also called quadratic loss and L2 loss, and is often used in regression prediction tasks. The mean square error function measures the quality of a model by calculating the square of the distance (ie, the error) between the predicted value and the actual value. That is, the closer the predicted value is to the true value, the smaller the mean square error between the two is.

Suppose there are $n$ training data$x_i$, each training data $x_i$The real output of is $y_i$，Model 对$x_i$The predicted value is ${\hat{y}}_i$. The model is in nnThe mean square error loss generated under$n$$$MSE=\frac{1}{n}\sum _{i=1}^n{\left(y_i-{\hat{y}}_i\right)}^2$$ Assume that the true target value is 100 and the predicted value is between -10000 and 10000. Draw the MSE function curve as shown in the figure below. It can be seen that when the predicted value is closer to 100, the MSE loss value is smaller. MSE loss ranges from 0 to$\infty$ 。