Machine Learning Notes (5) Logistic Regression, Decision Boundary, OvR, OvO

1. Classification problem

There are three major problems in machine learning, namely regression, classification and clustering. Linear regression belongs to regression tasks, while logistic regression and k-nearest neighbor algorithm belong to classification tasks. Logistic regression algorithms are mainly used in classification problems, such as the classification of spam (is spam or not spam), or the judgment of tumors (malignant tumor or not malignant tumor). In the problem of binary classification, we often use 1 to represent the positive category and 0 to represent the negative category.

2. Logistic regression

Logistic regression is generalized linear regression, so it has many similarities with linear regression. Their model forms are basically the same, both have $\hat{y}=wx+b$ , where w and b are the parameters to be requested. The difference is that their ranges are different. The range of linear regression is$[+\infty ,-\infty ]$ , the logistic regression range is$[0,1]$ . Logistic regression can be regarded as both a regression algorithm and a classification algorithm. It is usually used as a classification algorithm and can only solve binary classification problems. How does the regression problem solve the classification problem? As a classification task, we associate the characteristics of the sample with the probability of occurrence of the sample, and the probability is a number between [0,1]. put$\hat{p}\ge 0.5$ is classified as positive example 1, and the probability$p\; ^\; \le \; 0.5$$\backslash$$\hat{p}\le 0.5Indefinitely$ 0 that
$$\hat{p}=f(x)\hat{y}=\{\begin{array}{rlr}1,& & \hat{p}\ge 0.5\\ 0,& & \hat{p}\le 0.5\end{array}$$
So since logistic regression is inextricably linked with linear regression, how does it change the range of linear regression $[+\infty ,-\infty ]$ bound to
$[0,1]$ between a probability value? Hypothetical function y ^ = θ T x \hat y=\theta^Txin linear regression problems$\hat{y}=i^{T}x$, and want its output value to satisfy$0\le \hat{y}\le 1$ , introduce the sigmoid function, as follows:
$$sigmoid(x)=\frac{1}{1+e^{-x}}x=i^{The\; T}x$$
sigmoid function image is shown below. From the function image, the sigmoid function can meet the basic requirements of logistic regression, and its value range is$(0,1)$ at$x>When\; 0$ ,$\hat{p}>0.5$，在$x<When\; 0$ ,$\hat{p}<0.5$ .$\_$$\_$So the hypothesis function in a logistic regression problem:
$$\hat{p}=sigmoid({\theta }^T\cdot x_{)}=\frac{1}{1+e^{-i^T\cdot x}}\hat{y}=\{\begin{array}{rlr}1,& & \hat{p}\ge 0.5\\ 0,& & \hat{p}\le 0.5\end{array}$$

3. Loss function

• Representation of the cost function

For a given sample data set X,y, how do we find the parameters $\theta$ . As with the linear regression problem, find a suitable$The\; \theta$ parameter is very important for fitting the hypothesis function to the training set, and finding the minimum value of the loss function is undoubtedly crucial. So the question is, how should we represent the loss function. For logistic regression, we require the cost to meet the following requirements:
$$cost=\{\begin{array}{rl}& ify=1,the\; smallerp\; is,the\; largercost\\ & ify=0,the\; biggerp\; is,\end{array}$$
At this moment, do you think of the − log ( x ) -log(x) learned in high school?$-log\left(x\right)$ function at$(0,1)$ just meet our requirements, so we can express the cost as follows:
$$cost=\{\begin{array}{rlrlrlr}& & -log(\hat{p})& & if& & y=1\\ & & -log(1-\hat{p})& & if& & y=0\end{array}⟺cost=-ylog\left(\hat{p}\right)-(1-y)log(1-\hat{p})$$
according to$y=1$和$y=0$ two different cases, the image of the cost function is as follows:

It is easy to find that if y=1, for $-log\left(x\right)$ , the loss tends to ∞ \inftywhen taking 0$\infty$ , taking 1 means no loss. And if y=0, for$-log(1-x)$ , when taking 1, the loss tends to$\infty$ , taking 0 means no loss. At this point we get the cost function of logistic regression
$$J(\theta )=-\frac{1}{m}\sum _{i=1}^m{y}^{(i)}log(simgoid(X_b^{(i)}i\left)\right)+(1-y^{(i)})log(1-sigmoid(X_b^{\left(i\right)}i\left)\right)$$

• Minimization of the cost function
  Same as linear regression, after obtaining the cost function, you need to find the minimum value of the cost function, still using the gradient descent algorithm, the solution result is as follows:
  $$\frac{J(\theta )}{i_j}=\frac{1}{m}\sum _{i=1}^m(simoid(X_b^{(i)})-y^{(i)})X_j^{\left(i\right)}$$
  $$\nabla J\left(\theta \right)=\frac{1}{m}\cdot \left[\begin{array}{c}\sum _{i=1}^m({\hat{y}}^{(i)}-y^{(i)})\cdot X_0^{(i)}\\ \sum _{i=1}^m({\hat{y}}^{\left(i\right)}-y^{\left(i\right)})\cdot X_1^{\left(i\right)}\\ \sum _{i=1}^m({\hat{y}}^{\left(i\right)}-y^{\left(i\right)})\cdot X_1^{\left(i\right)}\\ \cdots \\ \sum _{i=1}^m({\hat{y}}^{\left(i\right)}-y^{\left(i\right)})\cdot X_n^{\left(i\right)}\end{array}\right]=\frac{1}{m}\cdot X_b^T\cdot (sigmoid(X_{b}i)-y)$$
  $$where{X}_b=\left[\begin{array}{ccccc}1& X_1^{(1)}& X_2^{\left(1\right)}& \cdots & X_n^{(1)}\\ 1& X_1^{(2)}& X_2^{\left(2\right)}& \cdots & X_n^{(2)}\\ \cdots & & & & \cdots \\ 1& X_1^{(m)}& X_2^{\left(m\right)}& \cdots & X_n^{(m)}\end{array}\right]i=\left[\begin{array}{ccccc}{i}_0& i_1& i_2& \cdots & i_n\end{array}\right]$$

4. Decision boundary

The so-called decision boundary is a boundary that can correctly classify samples, mainly including linear decision boundary and nonlinear decision boundary. Here is a chestnut:
for $sigmoid\left(x\right)$ , when$x>When\; 0$ , the predicted probability$p>0.5$ , the model classifies this class as 1, when$x<When\; 0$ , the predicted probability$p<0.5$ , the model class is divided into 0, the model is at$x=0$ successfully separated the categories. And for logistic regression$x=i^T\cdot x_b=0$ , the decision boundary is$i^T\cdot x_b=0$ , if X has two features:$$i_0+i_1{x}_1+i_2{x}_2=0\Rightarrow x_2=\frac{i_0-i_1{x}_1}{i_2}$$, we $x_{2}denotedbyy,x{}_{1}Expressx,ywith=\frac{i_0-i_{1}x}{i_2}$We can draw its decision boundary


nonlinear decision boundary

5. OvR and OvO

The logistic regression algorithm can only solve the binary classification problem, but we can use some strategies to make the logistic regression algorithm also able to solve the multi-classification problem. There are two strategies to allow algorithms that can only solve binary classification problems to solve multi-classification: OvR and OvO. Of course, these two methods are not only applicable to the logistic regression algorithm, but a general method that can transform almost all binary classification algorithms.

• OvR（One vs Rest）

Let’s first look at OvR (One vs Rest). From the English name of OvR, we can know that it is a pair of remaining possessions. However, in some other machine learning textbooks or materials, OvR may be called OvA (One vs All). The two express means the same. However, it is more accurate to call it OvR, and it is also named after OvR in the Sklearn documentation.
What is a pair of remaining all? For example, for the four classification tasks in the figure below.

Here, four different colors of points are used to represent four different categories. Obviously, for such a four-category task, the logistic regression algorithm that can only solve the two-category problem cannot be directly used. However, we can convert it into a two-category problem, and select one of the categories accordingly, for example, choose the red category at this time, and call them other categories for the remaining three categories, which is also the meaning of One vs Rest. The red category is One and all the remaining categories are Rest.

So far, the four-category problem has been converted into a two-category problem. Of course, the process of converting the two classifications not only includes the red category, but also the two classifications corresponding to the other three categories. Correspondingly, there will be four kinds of two classifications. The details are shown in the figure below.

• OVO（One vs One）

OvO (One vs One), through the English name of OvO, you can know that it means one-to-one. Here we still use the four categories with four categories as an example, and each category is represented by a sample point of a different color.
OvO is to directly select two of the categories each time. For example, the two categories of red and blue are picked out here, and then the two categories of red and blue are classified into two categories.
So far, the four-category problem has been converted into a two-category problem. For the four categories, this process can be repeated. Two categories are selected for each of the four categories. There are a total of C(4, 2) = 6 different pairwise category pairs, that is, 6 binary classification tasks are formed.
For 6 binary classification problems, each binary classification can estimate which of the two categories the new sample belongs to, and then the 6 classification results vote to select the category with the largest number of classification results as the category of the new sample point .

Reference:
https://cloud.tencent.com/developer/article/1605609