1. loss of function, cost function and objective function
Loss function (Loss Function) : is defined on a single sample, a sample refers to the error.
Cost function (Cost Function) : is defined over the entire training set, the average of all samples of the error, which is the average function value for all losses.
The objective function (Function Object) : refers to the need to optimize the ultimate function of general experience + risk structural risk, which is the (+ cost function regularization term).
--
2. The common loss function
(1) 0-1 loss function (0-1 loss function)
That is, when the prediction error is a loss of function, when the prediction is correct, loss function value is zero. The loss function without considering the degree of error of the predicted value and the true value. As long as the error is 1.
(2) quadratic loss function (quadratic loss function)
It refers to the difference between the predicted and actual values of the square.
(3) the absolute value of the loss function (absolute loss function)
Almost significance and above the loss function, just take the absolute value rather than absolute value, the gap is not enlarged square.
(4) Number of loss function (logarithmic loss function)
The loss function is more difficult to understand. In fact, the loss function uses the idea of maximum likelihood estimation. P (Y | X) popular explanation is: based on the current model, for sample X, its predictive value Y, which is the predicted probability of being correct. Between the probability of the simultaneous needs of a multiplication, in order to convert it to an adder, which we will logarithm. Finally, as is the loss function, so the higher the probability of correctly predicted its loss value should be smaller, so coupled with a negative sign to take a trans.
（5）Hinge loss
Hinge loss general classification algorithm loss function, in particular the SVM, which is defined as:
3. Common cost function
(1) the mean square error (Mean Squared Error)
Refers to the mean square error is the difference between a desired value and the parameter estimates of the true value of the square of the parameter; MSE data can be evaluated the degree of change, the smaller the MSE value is, the predictive model described the experimental data with better accuracy. (Ii represents ii sample, NN represents the total number of samples)
are generally used for the cost function regression problems .
(2) RMSE
RMS error is the square root of the mean square error arithmetic, to visually observe the degree of dispersion of the predicted and actual values.
Commonly used as a performance regression algorithm .
(3) the mean absolute error (Mean Absolute Error)
Mean absolute error is an average absolute error, mean absolute error can better reflect the actual situation of the prediction error.
Commonly used as a performance regression algorithm .
(4) the cross entropy cost function (Cross Entry)
Cross entropy is used to assess the probability distribution of the resulting difference in current training situation and the real distribution, reduce cross-entropy loss is to improve forecast accuracy in the model. Where p (x) p (x) is the probability that the true distribution, q (x) is the probability data computed by the model estimates.
For example, cross entropy cost function binary model (refer to a logistic regression):
Where f (x) f (x) may be a sigmoid function. Or other activation function of depth in the study. And y (i) ∈0,1y (i) ∈0,1.
Commonly used as a cost function classification problem.
