Python machine learning (5) logistic regression, decision boundary, cost function, gradient descent method to achieve linear and nonlinear logistic regression

The problem solved by linear regression is to introduce the characteristics of the data set into the model, predict a value to minimize the error, and the predicted value is infinitely close to the real value. For example, other characteristics of the house are introduced into the model to predict the housing price. The housing price is a series of continuous values, and linear regression solves supervised learning. The results predicted by many scenarios are not necessarily continuous, and the problem we want to solve is not always similar to the problem of housing prices.

classification problem

Whether the prediction is red blood cells or white blood cells, red blood cells and white blood cells are two completely different categories. When making predictions, you must first have historical data, train the model, and then evaluate the model repeatedly to get the ideal model, then pass the new data into the model, make a series of predictions, and get red blood cells (0), Or white blood cells (1), which is the simplest binary classification problem.

If linear regression is used to solve classification problems, $y=0$ is red blood cells,$y=1$ is white blood cells. The presentation of the data set is shown in the figure below. At this time, a line needs to be found to separate the two. If linear regression is used to do it, one must consider the minimum cost function (minimum error), and the other must be the best data separate. To separate red and white blood cells, take a value (0.5) on the line, if$h(x)>=If\; it\; is\; 0.5$ , the obtained point is above, and the predicted result is 1; if$h\left(x\right)<If\; it\; is\; 0.5$ , the obtained point is below, and the predicted result is 0.

If there is one more sample point in the data, as shown in the figure below, the solution of the fitting line should be the minimum cost function, and the fitting line will expand to the right, which is the blue line in the figure, if h ( x$h(x)=If\; it\; is\; 0.5$ , the points that will appear in the A area are not exactly 1. That is to say, when there is an abnormal sample point in the data, when the linear regression model is used to solve the problem, our overall prediction will change. At this time, the logistic regression algorithm must be introduced.

logistic regression

The logistic regression algorithm is one of the most popular and widely used algorithms today. Although its name contains the word regression, it is actually used to solve classification problems. Commonly used scenarios: data mining; automatic disease diagnosis; economic forecasting, and spam classification, etc. Logistic regression is also more important in deep learning. It is a relatively classic algorithm, and many of its principles are used in deep learning and neural networks.

Implementation of Logistic Regression

Prediction function: $h\left(x\right)=i^{T}X$, the predicted value will be far greater than 1, or far less than 0, and it will not be able to be classified. The goal:$h\left(x\right)$ converges between 0 and 1,$0<=h(x)0<=1$。
实现：使用$sigmoid(Logistic)$函数，$g(z)=\frac{1}{1+e^{-z}}$. Tozz $z$ tends towards$+\infty$，$e^{-z}$ tends to 0,$g\left(z\right)$ tends to 1 infinitely; when$z$ tends to$-\infty$，$e^{-z}$ tends towards$+\infty$，$g\left(z\right)$ tends to 0 infinitely.

Put $h\left(x\right)$ into$g\left(z\right)$ , get$g(i^{T}X)=\frac{1}{1+e^{-i^{T}X}}$, you can use $g\left(i^{T}X\right)$maps between 0 and 1.

When$i^{T}X>=0$ ,$g(i^{T}X)>=0.5$ , close to 1;
when$i^{T}X<0$ ,$g\left(i^{T}X\right)<0.5$ , close to 0. The formula can also be written as$h(x)=\frac{1}{1+e^{-i^{T}X}}$.
A set of data is trained to produce a model, new data is substituted into the model, and the predicted result is obtained. The result cannot be exactly 0 or 1, or it may be between 0 and 1. If the obtained result h ( x ) = $h\left(x\right)=0.7$ , it can be predicted that there is a 70% probability of white blood cells (1) and a 30% probability of red blood cells (0). Probability of being one of them:$h\left(x\right)=P(and=1|x;\theta )$ ,and$y=1$ condition$Probability\; of\; x$ ; sum of probabilities between:$P(and=1|x;i)+P(and=0|x;i)=1$。

• sum
  $h\left(x\right)$ using$g\left(i^{T}X\right)$converges between 0 and 1.
  $h(x)=g(i^{T}X)=P(and=1|x;\theta )$，$g(z)=\frac{1}{1+e^{-z}}$，
  当$i^{T}X>0,h(x)>=0.5$ , then predict$y=1$；
  当$i^{T}X<0,h\left(x\right)<0.5$ , then predict$y=0$

decision boundary

Help us better understand logistic regression and understand the connotation of function expression, $x_1$，$x_2$The distribution represents the feature, so $h\left(x\right)=g(i_0+i_1{x}_1+i_2{x}_2)$ , the second half is equivalent to$i^{T}X$.
As shown in the figure below, suppose$i_0=-3，i_1=1，i_2=1$ , get$i^{T}X=-3+x_1+x_2$, if $-3+x_1+x_2>=0$ , meaning$h(x)>=0.5$ , the value is closer to 1, then$y=1$ is more likely to be divided into 1; if$-3+x_1+x_2<0$ , meaning$h(x)<0.5$ , the value is closer to 0, then$y=0$ is more likely to be divided into 0. Draw the line according to the expression, will$-3$ moves to the right of the equal sign when$x_1=0$ ,$x_2=3$ ; when$x_2=0$ ,$x_1=3$ , draw a line between two points, above the line is$x_1+x_2>=In\; the\; part\; of\; 3$ , the category is predicted to be 1, and similarly, the category below the line is predicted to be 0.

In the figure below, positive samples are represented by a cross, and negative samples are represented by a circle. At this time, a straight line cannot be used to divide the two. In the previous linear regression, if the data cannot be fitted by a straight line, polynomial regression is used. Add some higher-order expressions. The same approach can be used for logistic regression as well.
$h(x)=g(i_0+i_1{x}_1+i_2{x}_2+i_3{x}_1^2+i_4{x}_2^2)$，whereθ$i_0+i_1{x}_1+i_2{x}_2+i_3{x}_1^2+i_4{x}_2^2$Equivalent to $i^{T}X$.
As shown in the figure below, suppose$i_0=-1，i_1=0，i_2=0，i_3=1，i_4=1$ , after substituting into the formula, we get$i^{T}X=-1+x_1^2+x_2^2$，若$-1+x_1^2+x_2^2>=0$ , then we can get$i^{T}X=x_1^2+x_2^2>=1$，$If\; the\; value\; of\; h\left(x\right)$ is greater than 0.5, the category is divided into 1, otherwise the category is divided into 0. $x_1^2+x_2^2=1$ is a standard circle with the origin as the center and a radius of 1, that is, the decision boundary. Points outside the circle are larger than the radius and belong to category 1. On the contrary, points inside the circle are smaller than the radius and belong to the category is 0. The decision boundary is through$\theta$ to determine,$h(x)=\frac{1}{1+e^{-i^{T}X}}$, 1 and e are constants, X is the data sample set (features), only $\theta$ is a parameter, as long as θ is determined$\theta$ also determines the decision boundary$h\left(x\right)$ , that is, the boundary h ( x ) h(x)can be predicted$The\; value\; of\; h\left(x\right)$ .

Solving for$\theta$ value, similar to linear regression, find$\theta$ is based on the cost function, so that the cost function is minimized to obtain$Theta$ value.

cost function

There is only one global optimal solution for a convex function. When a non-convex function seeks an optimal solution, it is likely to fall into a local optimal solution instead of a global minimum. Non-convex functions cannot achieve a global minimum by gradient descent.

The cost function defined by linear regression is: $J(\theta )=\frac{1}{2m}\sum _{i=1}^m(h(x^i)-y^i{)}^2$ , the sum of the squares of the real value minus the predicted value, and then divided by the number of features, which is the mean square error, at this time$h(x^i)=i_0+i_1{x}^i$ , if the cost function is applied to logistic regression, then$h\left(x^i\right)$is no longer a simple linear regression relationship, but$h(x)=\frac{1}{1+e^{-i^{T}X}}$, Substituting the content after the equal sign into the cost function as a whole, the graph will become a non-convex function, which is not convenient for finding the global minimum.
Goal: Find a different cost function such that $J\left(\theta \right)$ becomes a convex function.
Realization: Use logarithm to remove the impact of indexation, transform it into a linear relationship, and use logarithm to hedge the index. $2^n=4$ , can be converted to$lo{g}_{2}4=n$ , get$n=2$ .
Solution: Convert to a convex function. If it is unary, directly find the second order derivative. If it is greater than or equal to zero, it is a convex function; if it is multivariate, use the hessian matrix to solve it, which involves positive definiteness.
Function: The local optimal solution of the convex function is the global optimal solution.

• 当$y=When\; 1$ , the cost function$Cost(h(x),y)=-lo{g}_e(h(x))$
  $Costis$ the loss predicted by the current sample. P is the probability,$y=1$ probability.
  when$y=1$ ,$h\left(x\right)$ must be close to 1 to minimize the cost function,$y$ is the real value, the category is 1,$h(x)=P(and=1|x;\theta )$ is the probability that the predicted value is 1, the higher the probability, the closer to the result$y=1$ . If$h\left(x\right)=1$ is the best effect,$Cost=-lo{g}_e(h(x))=0$ , which means the loss is the smallest, and the cost function is 0;
  if$h(x)=0$，$h\left(x\right)=P(and=1|x;\theta )$ is the probability of the predicted value being 1 is 0,$Cost(h(x),y)=-lo{g}_e\left(h\right(x))$ is infinite and the loss value is very large.
  
• 当$y=When\; 0$ , the cost function$Cost(h(x),y)=-lo{g}_e(1-h(x))$
  当$y=0$ ,$h(x)$为1，$Cost\left(h\right(x),y)=-lo{g}_e(1-h(x))$ has a predicted value of 1 with probability 0,$log(1-h(x))$ is infinite; otherwise$h\left(x\right)$ is 0, tends to$y=0$ category,$-lo{g}_{e}1=0$ , the loss is minimal.
  
  Note: For logistic regression, it is not necessary to distinguish between predicted probability categories. When$h(x)=P>=0.5$ , divided into the category of 1, close to 1; when$h\left(x\right)=P<0.5$ , divided into the category of 0, tends to 0.
  The above two cost functions are segmented, solving$\theta$ value to solve the actual problem, and$y=0or1$ , you can simplify the equation to find the cost function, and integrate the above two formulas into one, which is convenient for later derivation. $$Cost(h(x),y)=-ylo{g}_e(h(x))-(1-y)lo{g}_e(1-h(x))$$
  
  $Cost\left(h\right(x),y)$ is the loss of a sample data, each sample point has a loss, it is necessary to integrate many sample points, sum and divide by the number of samples, put forward the negative sign, and get the following formula, this The method is also cross entropy.
  $$J(\theta )=\frac{1}{m}\sum _{i=1}^{m}Cost(h(x),y)=-\frac{1}{m}\sum _{i=1}^m[ylo{g}_e(h(x))+(1-y)lo{g}_e(1-h(x))]$$
  Logistic regression is a very commonly used algorithm and is also used in deep learning. This equation uses the maximum natural method in statistics to quickly find parameters for different models. It is also a convex function, which solves the previous problem of non-convex functions and facilitates the next derivative.

Gradient descent method derivation

cost function：
$$J(\theta )==-\frac{1}{m}\sum _{i=1}^m[ylo{g}_e(h(x))+(1-y\left)lo{g}_e\right(1-h(x))]$$
Goal: Find$\theta$ , so that the cost function$J\left(\theta \right)$ is the smallest.
Substitution method,$z=i^{T}X$,对$The\; derivative\; of\; g\left(z\right)$ becomes$\frac{1}{1+e^{-z}}$Take the derivative, $1+e^{-z}$ derivation becomes$e^{-z}$ derivative multiplied by$-$ Derivative of$z$$z=i^{T}X$derivation, X is a constant,$i^T$ is a variable, after derivation is$x$

The result after the derivation of the cost function is:

the essence of the gradient descent method is to iterate the direction of the curve descent through non-stop derivation. point, the process with the smallest loss
$lo{g}_{e}x=lnx=\frac{1}{x}$

Gradient Descent for Linear Logistic Regression

sklearn implements linear logistic regression

Logistic Regression API

sklearn.linear_model.LogisticRegression(solver='liblinear',penalty='l2',C=1.0,
solver 可选参数：{
    
    'liblinear','sag','saga','newton-cg','lbfgs'}
penalty：正则化的种类
C：正则化力度

liblinear is the default value, which is an algorithm for optimization problems and is suitable for small data sets; sag and saga are used for large data sets, and newton-cg is used for multi-class problems.
There are positive and negative examples in the data, and the sklearn interface defaults to positive examples with a small number.

Gradient Descent Method for Nonlinear Logistic Regression

Taxonomy Assessment Report API

sklearn.metrics.classification_report(y_true,y_pred,labels=[],target_names=None)
y_true：真实目标值
y_pred：估计器预测目标值
labels：指定类别对应的数字
target_names：目标类别名称
return：每个类别精确率与召回率

Use the dataset provided by sklearn