Loss function - mean square error & cross entropy

Mean square error:

 

 cross entropy 

Binary classification cross entropy

 The base of log in the expression is e

yi represents the label of sample i. Positive is 1 and negative is 0.

pi represents the predicted probability of sample i

Multi-class cross entropy

 M: indicates the number of categories

yic: sign function (0 or 1), if the true category of sample i is equal to C, it takes 1, otherwise it takes 0

pic: The predicted probability of class C to which sample i belongs.

 

Summarize:

Let’s use the output of the last layer of the neural network to take a look at the entire model prediction, loss and learning process:

1. The last layer of the neural network obtains scores (also called logits) for each category ;
2. The score is passed through the sigmoid (or softmax) function to obtain the probability output;
3. The class probability output predicted by the model and the one hot form of the true class are used to calculate the cross entropy loss function.

reference

https://zhuanlan.zhihu.com/p/35709485