Lasso regression series two: the principle of Lasso regression/ridge regression

The principle of Lasso regression/ridge regression

When learning the role and difference of L1 and L2 regularization, we always see such a picture:

This picture visually explains the different constraint effects of L1 and L2 on the linear model.

At first, I didn't really understand why I drew like this. for example

1. Will the L1-norm contour line and the square error term contour line intersect on a certain coordinate axis?

2. Can Lasso regression only use the sum of squared errors as a loss, can it be replaced by cross entropy?

3. In addition to L1-norm and L2-norm, are there other regularization methods? What is the difference between them?
See my other blog Lasso regression series three: L0, L1, L2, L2,1 norms in machine learning

Now I have figured it out, combined with several good blogs on the Internet, I will sort it out and share it with you. If there are any deficiencies or mistakes, please correct me.

overview

The regression model using the L1 regularization term is called Lasso regression (Lasso Regression), and the regression model using the L2 regularization term is called Ridge Regression.

Therefore, as long as the L1 regular term is added to the regression problem, it can be called Lasso regression, and it is not limited to the case of using the sum of squared errors as the loss.

In this article, first, we will understand what changes will happen to the solution when adding L1-norm Lasso regression and L2-norm ridge regression when using least squares estimation to solve linear regression problems, so as to better understand how to use Lasso regression and ridge regression.

Least Squares Estimation of Linear Models

When estimating parameters for linear models, the method of least squares can be used.

Described in mathematical language, the linear model can be expressed as:
$$\begin{gathered} y=X\beta +ϵ \\ E\left(ϵ\right)=0,Cov\left(ϵ\right)=p^{2}I \end{gathered}$$
where$y$ is$n\times$label vector of$1$$X$ is$n\times The\; characteristic\; matrix\; of\; p$ (corresponding to the data,$n$ is the number of samples,$p$ is the number of features),$\beta$ ,$ϵ$ is the parameter to be estimated,$\beta$ is$p\times$Unknown parameter vector of$1$$ϵ$ is the random error.

The least square method is to estimate the parameter vector $The\; basic\; method\; of\; \beta$ , the idea is to make the error as small as possible, that is, $\epsilon = y- X\beta $ as small as possible, that is, to make Q ( β ) = ∣ ∣ ϵ ∣ ∣ 2 = ∣
$$Q\left(b\right)=||ϵ|{|}^2=||y-X\beta |{|}^2=(y-X\beta {)}^T(y-X\beta )$$
should be as small as possible.

According to the theorem that the minimum value of a convex function is the minimum value, we can obtain the β \beta where the partial derivative is equal to 0$\beta$ value, so that the above formula reaches the minimum value, namely:
$$\hat{b}=(X^{T}X{)}^{-1}{X}^{T}y$$
combines the knowledge in matrix theory, when$rank(X)=p$ ,$X^{T}X$is reversible, then$\beta$ has a unique solution, $\hat\beta = \beta $, it is said that $ \hat\beta $ is an unbiased estimate of $ \beta $; when$rank\left(X\right)<p$ ,$The\; X$ matrix is ​​not full of rank. At this time, we cannot get an unbiased estimate of $ \beta $, which leads to$rank\left(X\right)<There\; are\; generally\; two\; reasons\; for\; p$ : 1. The number of samples is smaller than the number of features. 2. Even if the number of samples is large, there is a linear relationship between variables (features). Lasso regression and Ridge (ridge regression) are used to solve this problem. of.

Lasso regression and Ridge regression

Lasso regression (lasso regression) is to add a weight β \beta after the objective functionThe 1-norm of$\beta$
$$\begin{gathered} Q\left(b\right)=||y-X\beta |{|}_2^2+\lambda ||\beta |{|}_1 \\ ⟺ \\ \arg \min ||y-X\beta |{|}^{2}s.t.\sum |{\beta }_j|\le s \end{gathered}$$

Ridge regression (ridge regression) is to add a weight β \beta after the objective function$\beta$的2-范数，即：
$$\begin{gathered} Q\left(b\right)=||y-X\beta |{|}_2^2+\lambda ||\beta |{|}_2 \\ ⟺ \\ \arg \min ||y-X\beta |{|}^{2}s.t.\sum b_j^2\le s \end{gathered}$$

Solving the above formula, we can get $\beta$的岭可以：
$$\hat{b}\left(l\right)=(X^{T}X+I)\_{}^{-1}{X}^{T}y$$
ensures that$X^{T}X+\lambda I$ is full rank and reversible. Of course,$\hat{b}\left(\lambda \right)$ is a biased estimate.

Why is Lasso regression easier to generate sparse solutions?

Let's look at this picture again:

The square error contour is $Q\left(b\right)=||y-X\beta |{|}^2$ Corresponding equipotential lines

Lasso regression corresponds to the L1 norm contour, and Ridge regression corresponds to the L2 norm contour, both of which pass the regularization parameter $\lambda$ to adjust the parameter$The\; degree\; of\; constraint\; of\; \beta$ .

Lasso regression is easy to produce sparse solutions because the L1 norm contains some non-differentiable corners on the coordinate axis (non-differentiable corners), these corners and $Q\left(b\right)=||y-X\beta |{|}^{The\; probability\; of\; 2}$ intersecting will be much higher. In Ridge regression, the L2 norm is differentiable everywhere, so sum$Q\left(b\right)=||y-X\beta |{|}^2$ The probability of intersecting on the coordinate axis will be much smaller.

In addition, for the L1 norm, $The\; larger\; the\; \lambda$ ,$||\beta |{|}_1$The smaller the range, the greater the probability that the square error contour and the L1 norm contour intersect on the coordinate axis, that is to say β \betaThe greater the probability that the elements in$\beta \; become\; 0.$Conversely,β \betaThe probability that the elements in$\beta \; become\; 0\; is\; smaller.$

reference

L1, L2 regularization method

Lasso—principle and optimal solution