Thorough understanding of regularization (Regularization)

First, we must understand the loss function visualization. For

the loss function contour map in the parameter space, there are infinitely many solutions that meet the same loss value .

The general form of adding regular terms to the loss function:
$$L=\sum _{i=1}^n[{and}_i-\sum _{j=1}^p(w_j\ast x_i)-b]+\lambda \sum _{j=1}^p|w_j{|}^q$$

We make ${\sum }_{j=1}^p|w_j{|}^q$ in differentqqThe image under$q$ :

In theloss function visualization, we drew the loss function${\sum }_{i=1}^n[{and}_i-{\sum }_{j=1}^p(w_i\ast x_i)-b]$ contour form, if the loss function image without the regular term and the regular image are put together:

Take the outermost circle of the left picture, when the loss function value reaches the value corresponding to the outermost circle, then$w_1$And $w_2$There are infinitely many, so if we add $l_1$Regular term, which means to choose one from the infinite number, this one is from the infinite number $w_1+w_2$The smallest one. If a straight line intersects multiple contour lines, then w 1 + w 2 w_1+w_2 at these intersections$w_1+w_2$They are all equal. At this time, the solution with the smallest contour is selected, as shown in point 5 in the figure.

The picture on the right is the same $l_2$Regular w 1 w_1 corresponding to the point tangent to the contour for the first time$w_1$And $w_2$Is an infinite number of solutions $w_1^2+w_2^2$The smallest one.

So it can be concluded that the first intersection of the contour and the regular term is the optimal solution. The regular term reduces the scope of the parameter space we solve.

https://blog.csdn.net/zandaoguang/article/details/107970123
http://freemind.pluskid.org/machine-learning/sparsity-and-some-basics-of-l1-regularization/#ed61992b37932e208ae114be75e42a3e6dc34cb3http://

In-depth understanding of regularization from a Bayesian perspective
-must see

Why not use L0 as a regular term?
From a theoretical point of view, L0 is indeed the best regular term for sparse solutions, but the dimension of features in machine learning is often very large. You can also understand that there are many coefficients, and then solving L0 is an NP-hard problem again, so in It is extremely limited in practical engineering applications and is not feasible.

Why do we have to get a sparse solution?
This question is not absolute. Statistically speaking, sparse solution can alleviate the overfitting problem of the model. After all, it can reduce the complexity of the model-that is, make some attributes invalid.