Chapter 1: part3 classification model

The value of the classification learning output is selected from several possible values , but this method cannot continue to use linear regression, and a new model is required: logistic regression

1. Introduction to binary classification problems

Binary classification model

Many questions dictate answers:

For example: the answer below is only yes/no (1/0)

For the above problem, the dataset used to represent:

1: It is a tumor

0: Not a tumor

• It is obviously unreasonable to use linear regression for prediction

The following will describe how to process the above data to get a desired result

logistic regression model

$$f_{w,b}(x)=g((\vec{w}\cdot \vec{x})+b)=\frac{1}{1+e^{-(\vec{w}\cdot \vec{x})+b}}$$

Since our output is in the interval of **0(no)-1(yes)**, we want to get a function, and the output is also 0-1:

• logic function

First let's look at the function that builds the logistic regression

$$\begin{gathered} g(z)=1/1+e^{-z} \\ It\; tends\; to\; 1\; whenzis\; large,\; and\; tends\; to0\; whenzis\; small \end{gathered}$$

• The establishment of the logistic regression algorithm

1. Substituting the straight line linear regression model into

The linear regression model is set to z substitution

2. Substitute into the above function

The function is a logistic regression model

In this way, the results in the linear regression equation can be in the range of 0-1

Example analysis of function output results:

例如：
有一个病人，x为肿瘤大小，y输出为肿瘤结果的概率（0-1），则下述的例子为：有70 %的概率患病

Combined with the knowledge of probability theory, it is usually possible to write:

Indicates the probability of y=1 in the case of x (corresponding to w and b)

decision boundary

If we only want 0/1 results, we can set the threshold

For example:

$$ f(x)= \begin{cases} 0,& \text{if x >0.5} \\ 1, & \text{if x <0.5} \end{cases} $$

• definition

When x=0.5, it means w*x+b=0 (as shown below)

It also means that f(x) has the same probability of being 0 and 1

What is the significance of such a fitting function, as follows:

例：训练集如下所示，x为1，o为0，假设函数为回归函数，有两个维度，w1=1，w2=1，b=-3

The decision boundary can be expressed as:
$$\begin{gathered} z=\vec{w}\cdot \vec{x}+b=0 \\ Substitute\; intow1,w2,b \\ {x}_1+x_2-3=0 \\ {x}_1+x_2=3 \end{gathered}$$

This purple line is the decision boundary

This is the basic function of the regression model. Let's introduce how to train the regression model in detail.

• build cost function
• Implementing the Gradient Descent Algorithm

---

2. Cost function in logistic regression

Cost Function Implementation in Logistic Regression

If using the previous cost function:
$$J(w,b)=\frac{1}{2m}\sum _{i=0}^m(f_{w,b}(x^{(i)})-y{)}^2$$

There will be many local minimum points, which is not convenient to use

• Optimize the cost function:

Put 1/2 inside the summation function

Using a convex function like this guarantees the use of the gradient descent algorithm

• Loss function L

Can tell us how well the model is trained on the sample

If the value of y is 0, the loss function is:

understand:

If the value of y is 1 , the loss function is:

$$-log(f):0<f<1$$

That is, when the threshold is f(x), the result is regarded as y=1 , what is the corresponding loss value

If the value of y is 0 , the loss function is:

$$-log(1-f):0<f<1$$

That is, when the threshold is f(x), the result is regarded as y=0 , what is the corresponding loss value

After building the loss function, then if you can know the w&b that makes J(w,b) the smallest , then you can implement logistic regression

Simplify the cost function

The above formula can be simplified into one formula:

Then substitute the above formula into the cost function

得到最优式
$$J(w,b)=-\frac{1}{m}\sum _{i=0}^m[y^{(i)}log(f_{w,b}({\vec{x}}^{(i)}))-(1-y^{(i)})log(1-f_{w,b}({\vec{x}}^{(i)}))]$$

This cost function is inferred based on the maximum likelihood estimation , and those who are interested can understand it by themselves

---

3. Implement Gradient Descent

Through the gradient descent algorithm, we can find the w&b that minimizes J(w,b)

The partial derivatives of the above formulas are calculated separately:

Substituting partial derivatives, the following formula is obtained:

Note: Here fw,b and different
$$f_{w,b}(x)=\frac{1}{1+e^{-(\vec{w}\cdot \vec{x})+b}}$$

---

4. Overfitting problem

Introducing overfitting

• Terminology introduction

• Generalization: the ability of the model to reason about unknown samples

• Underfitting (underfit)

The following is the model of the price

There is a bias in fitting a set of data , so it can also be called: high bias (high bias)

• Overfitting (overfit)

For a set of data that fits too well and loses predictive power , it can also be called: high variance (high variance)

So, we should look for a model that has neither high bias nor high variance

Solve the overfitting problem

1. more training data

2. Observe if you can use fewer features (x1, x2, x3...)

Disadvantage: some important features may be discarded

3. Regularization

narrow the value of the parameter as much as possible

Keep all features, but prevent feature weights from being too large

Regularization

Regularization only considers the influence of w , not b (b has less influence on the overall fitting)

• principle

If the binomial model is normal

But the tetranomial model overfits

We need to reduce the size of w3 and w4 to prevent the model from overfitting

So we can add w3 and w4 after the cost function :

In order to minimize the cost function, w3 and w4 must be given small values

• official

λ: regularization parameter, which needs to be given manually

λ cannot be too large ( all items tend to 0 ), nor can it be too small ( useless )

1. Regularized linear regression

official:

Substitute the gradient descent algorithm into:

Regularize only w

2. Regularized logistic regression

Overfitting Logistic Regression Plot

official:

Substitute the gradient descent algorithm into:

Regularize only w

**Note:** Here fw,b and different
$$f_{w,b}(x)=\frac{1}{1+e^{-(\vec{w}\cdot \vec{x})+b}}$$