What is the objective function?
The definition is: refers to the functional relationship between the target concerned and related factors. For example, if we want to predict how many hand cakes the shop downstairs will sell tomorrow, we know how many cakes it sells every day in the past 10 days, the weather conditions every day, whether there are holidays, and how many cakes the shop owner will sell. The relationship status of the proprietress, these data have been calculated. The explanation corresponding to the objective function is:
- The goal of concern is how many hand-grabbed cakes can be sold tomorrow, the amount is unknown.
- The relevant factors are the daily quantity, weather conditions, holidays, the emotional status of the boss and the wife, and known quantities.
- Objective function: Use these known conditions according to certain ideas to solve the functional relationship of unknown quantities.
loss function
Also called the cost function, it's the same thing. is the difference between the predicted value f(x) of the model and the true value Y, which is a non-negative real-valued function. Wait a minute, why is it non-negative? For example: For example, the hand grabbing cake mentioned above, the model predicts that the first day is 30, the real value is 29, and the gap between 30 and 29 is 1; the second day is predicted to be 35, the real value is 36, and the gap between 35 and 36 is - 1; If you add them directly, it is 1-1 = 0, is there no difference? It is very obviously wrong. If the gap is 0, people think that they are all right, but in fact they are all wrong, so there can be no negative values. You can use absolute values, squares or some other mathematical operations.
Common loss functions are:
- LR logistic regression: log logarithmic loss function reference: https://xiaozhuanlan.com/topic/7491865230
- Least Squares: Squared Loss Function Reference: https://xiaozhuanlan.com/topic/3129560874
- Adaboost: Exponential loss function (this algorithm will be introduced later)
Ok, now let's look at the usual form of the definition of the objective function:
Obj (Θ) = L (Θ) + Ω (Θ)
The Chinese translation is:
Objective function = loss function + regular term
We understand both the objective function and the loss function, so what does the regular term do?
Regularization
Going back to the example we gave earlier, if someone wins the lottery on the fifth day and buys 100 hand-pulled cakes excitedly, which is more than a few days, obviously this situation will affect our model training. This situation is overfitting. The purpose of adding a regular term is to prevent the model from overfitting.
Commonly used are L1 and L2 regularization, which are very familiar, right? The L1 and L2 norms in the linear algebra mentioned earlier are somewhat similar. In fact, the essence is the same. In machine learning, it is called regularization, linear algebra is called norm, and in statistics, it is called penalty term.
In machine learning: L1 uses absolute distance, also called Manhattan distance, L2 uses square distance, also called Euclidean distance
Linear algebra: The L1 norm calculates the sum of the absolute values of all elements of the vector, and the L2 norm calculates the vector length reference in the usual sense: https://xiaozhuanlan.com/topic/4961780352
Summarize:
The function of objective function, loss function and regular term is introduced. Optimizing the loss function makes the model avoid underfitting and minimizes the loss function, and optimizing the regular term makes the model avoid overfitting.