Machine Learning Machine Learning: Difference and connection between cost function, loss function, and objective function?
Regarding this issue, I am also a beginner in machine learning, and I feel that it is necessary to get to the bottom of these concepts.
Different authors have different opinions. By collecting some relevant author's answers and personal understandings, I have sorted out the materials as follows:
First, in Andrew NG's Coursera: Neural Networks and Deep Learning course, there is the following passage:
The loss function computes the error for a single training example; the cost function is the average of the loss funcitons of the entire training set.
The classroom PPT is as follows:
loss(error) function is the loss/error of a single sample; and cost function is the error description of the entire data set, which is the cost to be paid for estimating the data after selecting parameters w and b, cost It is obtained by averaging the errors of all data.
For a detailed explanation of this block, refer to the following explanation in Mr. Zhou Zhihua's book "Machine Learning".
Taking SVM as an example, the optimization objective function in SVM is as follows:
The empirical loss (loss) is the legendary loss function or cost function. The structural loss (Ω) is a function such as a regularization term that controls the complexity of the model.
Take a chestnut:
"Machine Learning" Objective Function Analysis
Among them, empirical risk is used to describe the degree of fit between the model and training data; C is used to make a compromise between the two analyses. From the perspective of empirical risk minimization, it expresses what kind of model we need to obtain (such as obtaining a model with less complexity), which provides a path for introducing domain knowledge and user intent; on the other hand, this information Helps to reduce the hypothesis space, thereby reducing the risk of overfitting to minimize training error.
