Regularization and normalization

The two concepts of regularization and normalization were often mistaken before, and this article is featured to prevent mistakes again.

1. Normalization

The function of normalization is to remove the dimension of the data , or to convert the value of the data to the same order of magnitude or limit it within a certain range.

1.1 max-min normalization

i.e. through xxThe maximum and minimum value pair xxof the data set where$x\; is\; located$$x$ is normalized:
$$x^{{}^{\prime }}=\frac{x-x_{\min }}{x_{\max }-x_{\min }}$$
Among them, $x_{\min }$and $x_{\max }$for data $The\; minimum\; and\; maximum\; values\; of\; the\; set\; (row/column)\; where\; x$ is located, after normalization,$The\; range\; of\; x$ is,$x\in [0,1]$。

1.2 Normalize with mean and variance (standardization)

Put the data $Transform\; x$ to mean 0 and variance 1:
$$x^{{}^{\prime }}=\frac{x-}{p}$$
Among them, $\mu$和$\sigma$ are the mean and variance of the data set, respectively.
After such normalization, the corresponding loss function has a uniform contour shape, which can quickly converge when performing the gradient descent algorithm.

2. Regularization

Regularization is mainly used to avoid overfitting and reduce network errors. The regularization formula is:
$$L=\sum _n{\left({\hat{y}}^n-\left(b+\sum w_i{x}_i\right)\right)}^2+l\sum {\left(w_i\right)}^2$$

Note 1: The formula comes from Professor Li Hongyi's 2020 machine learning courseware
Note 2: Commonly used L2 regularization

where, ${\hat{y}}^n$ represents the$The\; true\; value\; of\; n$ pieces of data,$x_i$for $i$ input features. Compared with the general loss function formula, it can be seen that regularization means adding a term$l\sum {\left(w_i\right)}^2$ , where$l\ge 0$ is used to adjust the degree of regularization.

Note: Usually the coefficient $w0$ is omitted from the regularization term because$w0$ will make the result dependent on the choice of the origin of the target variable

This formula adds a penalty term to the error function such that the coefficient $w_i$will not reach a large value.
The following table visually shows $The\; effect\; of\; \lambda$ on the coefficients:
$$\begin{array}{rrrr}& \ln l=-\infty & \ln l=-18& \ln l=0\\ w_0^{\ast }& 0.35& 0.35& 0.13\\ w_1^{\ast }& 232.37& 4.74& -0.05\\ w_2^{\ast }& -5321.83& -0.77& -0.06\\ w_3^{\ast }& 48568.31& -31.97& -0.05\\ w_4^{\ast }& -231639.30& -3.89& -0.03\\ w_5^{\ast }& 640042.26& 55.28& -0.02\\ w_6^{\ast }& -1061800.52& 41.32& -0.01\\ w_7^{\ast }& 1042400.18& -45.95& -0.00\\ w_8^{\ast }& -557682.99& -91.53& 0.00\\ w_9^{\ast }& 125201.43& 72.68& 0.01\end{array}$$
It can be seen that $When\; \lambda$ is small, the parameters of the model are very large, which can easily lead to overfitting of the model. With$When\; \lambda$ gradually increases, the coefficient will become very small again, which is not conducive to the fitting effect of the model. $\lambda$ controls the complexity of the model and determines the degree of overfitting.