Machine learning logistic regression

Basic definition

Binomial logistic regression, referred to as logistic regression, also known as logarithmic regression

Is a two-class model

Using probability to study the relationship between categories and features is a kind of nonlinear regression.

Introduction of log probability function

Because the output label of the binary classification task y ∈ {0, 1} y ∈ \{0,1\}Y∈{ 0,1 } , and the predicted value of linear regression$with=w^{T}x+b$ is a real value.

We want to convert the real value z to a value of 0/1. The most ideal is the unit step function:

because it is not continuous, we hope to find a substitute function that is close to it to a certain extent, and hope that the substitute function is monotonically differentiable. So the logarithmic probability function was introduced

$$Y=\frac{1}{1+e^{-with}}$$

If the predicted value z is greater than zero, it is judged as a positive type, if it is less than zero, it is judged as a negative type, and if it is a critical value of zero, it can be judged at will:

Derivation of log probability function

Let y be the probability that the sample x is a positive class, then 1-y is the probability that the sample x is a negative class

Their ratio $\frac{and}{1-and}$Called the probability , expressed as the relative probability of the positive class

Take the logarithm to get the logarithmic probability: $ln\frac{and}{1-and}$

令$with=ln\frac{and}{1-and}$， 可 得$Y=\frac{1}{1+e^{-with}}$

The significance of the log probability function : the probability of predicting the sample x as a positive class

Logistic regression model

将 $with=w^{T}x+b\; is$ substituted into the logarithmic probability function to obtain the form of the logistic regression model:

$$h=\frac{1}{1+e^{-(w^{T}x+b)}}$$

Estimation of model parameters

Cost function:

$$J=-\frac{1}{n}\sum _{i=1}^n{Y}_{i}lnh(x{}_i)+(1-Y_i)ln[1-h(x_i)]$$

Minimize the cost function to obtain the parameters w and b

Solution method: small batch gradient descent method