Machine learning linear regression analysis

Basic form of linear model

Sample predictions through linear combinations of attributes:

$$f(x)=w_1{x}_1+w_2{x}_2+...+w_d{x}_d+b$$

Written in the form of a vector:

$$f\left(x\right)=w^{T}x+b$$

w represents the weight of each attribute, b is the bias value, x is the sample vector, and f(x) is the predicted value

Linear regression

Regression analysis is a predictive modeling that studies the relationship between independent variables and dependent variables

Mathematical description :

Yi I Collection {(xi, yi), i = 1, .. ., n}, x ∈ RP, yi ∈ R \ {(x_i, y_i), i = 1, ..., n \}, x ∈ R ^ P, y_i ∈ R{ (xi​,Yi​),i=1,...,n},x∈RP,Yi​∈R

其中$Y_i=f(x_i)+{ϵ}_i$

${ϵ}_i$Means to $Y_i$ Prediction error

Solving parameters w and b

Generally speaking, we want to minimize the mean square error of the predicted value, and the parameters at this time are the parameters we want

Cost function:

$$J(w,b)=\sum _{i=1}^n{ϵ}_i^2=\sum _{i=1}^n({and}_i-f(x_i){)}^2$$

Linear regression models are trained using the smallest multiplication method.

Least squares criterion : The sum of squares of prediction residuals of each training sample is the smallest.

By minimizing the cost function, w and b are obtained:

$$[w^{\ast },b^{\ast }]=argminJ(w,b)$$

Common parameter solving methods

1. Analytical method

Find the partial derivative of the function, and then set the partial derivative to 0 (but you may encounter the case where the matrix is ​​not invertible).

Suitable for small samples

2. Numerical optimization method (gradient descent method, etc.)

Iterative solution using methods such as gradient descent

Suitable for situations with a large number of samples

Batch and mini-batch algorithms

1. Batch gradient descent method: use all training samples to estimate the gradient for training, which requires a large amount of calculation

2. Mini-batch gradient descent method: use partial training samples to estimate the gradient for training

3. Stochastic gradient descent method: each time a training sample is drawn from a fixed training set to estimate the gradient for training.

Regularization (parameter norm penalty)

By benchmarking the cost function $J$ adds a parameter norm penalty to limit the learning ability of the model. The overall cost function after regularization is:$$J^{{}^{\prime }}(w,b)=J(w,b)+\lambda \Omega (w)$$

Ω(w) represents the penalty item

L1 regularization (lasso regression) : introduce a norm penalty of parameters in the cost function,

L2 regularization (ridge regression) : introduce the two-norm penalty of parameters in the cost function,