Logistic regression of study notes

Introduction :
Today we learn logistic regression. We all know that the linear regression model is $y=w^TX+b$ , we transform him, get $lny=w^TX+b$ , this is "logit linear regression" (logit linear regression), which is what we call logistic regression. Deform again $y=e^{w^TX+b}$, generally, put $y=f(w^TX+b)$ Form, called generalized linear model.

1. Mathematical principles

Because X is continuous, so $w^TX+b$The result of$b$ is also continuous. In order to be able to do the binary classification problem, we draw on the unit step function. As shown below:

we introduce the sigmoid function, the form of logistic regression is $y=\frac{1}{1+e^{-z}},z=w^TX+b$ . We let y ≥ the threshold, the result is a positive example, otherwise, it is a negative example (the threshold is generally 1/2).
Note: Logistic regression draws on probabilistic thinking, but it is not a true probability, so it is different from Bayesian algorithm.

Loss function: $J(θ)=-\frac{1}{m} \sum_{i=1}^{m}y^{(i)}log(σ(θ^T・X_b^{(i)}))+(1-y^{(i)})(1-log(σ(θ^T・X_b^{(i)})))$ , When we look for the optimal logistic regression model, we are to find a set of parameters θ, so that the loss function J (θ) reaches the minimum value.

Second, the code implementation

1. Manual code implementation The
implementation process:
①, define the sigmoid method, use the sigmoid method to generate the logistic regression model;
②, define the loss function, and use the gradient descent method to obtain the parameters;
③, substitute the parameters into the logistic regression model to obtain the probability;
④. Convert probability into classification.

在这里插入代码片

2. Sklearn implementation In the
previous article, we learned regularization, which can improve the generalization ability of the model. sklearn's logistic regression algorithm is directly added to the regularization, the parameter is penalty, and the default is L2. In addition, a hyperparameter C is added. That is, the loss function form is $J = CJ (θ) + αL1 或 J = CJ (θ) + αL2$

References: "Machine Learning Zhou Zhihua" Chapter 3 Linear Model Section 3.2, Section 3.3
Reference Article:
https://mp.weixin.qq.com/s/xfteESh2bs1PTuO2q39tbQ
https://mp.weixin.qq.com/s / nZoDjhqYcS4w2uGDtD1HFQ
https://mp.weixin.qq.com/s/ex7PXW9ihr9fLOjlo3ks4w
https://mp.weixin.qq.com/s/97CA-3KlOofJGaw9ukVq1A
https://mp.weixin.qq.com/sg/BUX9Q9