[Machine Learning] Logistic Regression (Rearranging)

Logistic regression learning circuit

Preliminary knowledge: It is recommended to go to Station B to learn the concepts of information volume, entropy, BL divergence, and cross entropy.

Recommend Bilibili video: How does "cross entropy" work as a loss function? Package understanding of "information volume", "bit", "entropy", "KL divergence", "cross entropy"

Information refers to the amount of information contained in an event, usually represented by a logarithm with base 2. For example, if the probability of an event occurring is 1/8, then the amount of information of this event is log2(1/8)=-3, because three bits are required to represent it.

Entropy is a measure of uncertainty in a system or source, and can also be understood as the average amount of information. In information theory, greater entropy means that the system or source is less predictable and therefore contains more information. For example, a pile of coins that come up heads has many possibilities, so they have higher entropy than a pile of coins that are known to all come up heads.

KL divergence (Kullback-Leibler
divergence), also known as relative entropy, is a way to measure the difference between two probability distributions. The KL divergence is non-negative and takes a value of 0 if and only if the two distributions are identical.

Cross-entropy (Cross-entropy) is a method used to compare the difference between two probability distributions, which is often used to evaluate the performance of classification models. Similar to KL divergence, cross-entropy is also non-negative, taking a value of 0 if and only if the two distributions are equal.

Knowledge map for this section

Functional Model of Logistic Regression

 Logistic regression is a classification model

 It can be used to predict whether something will happen or not. Classification problems are the most common problems in life:

• In life: such as predicting whether the Shanghai Composite Index will rise tomorrow, whether it will rain in a certain area tomorrow, whether the watermelon is ripe

• Financial field: Whether a certain transaction is suspected of violating regulations, whether a certain company is currently violating regulations, and whether there will be violations in the future

• Internet: Will users buy something, will they click something

 For known results, the answer to the above question is only: 0, 1 .

 Let's take the following binary classification as an example. For a given data set, there is a straight line that can divide the entire data set into two parts:

 At this point, the decision boundary is $w_1{x}_1+w_2{x}_2+b=0$ , at this point we can easily put$$h(x)=w_1{x}_1+w_2{x}_2+b>A\; sample\; of\; 0$$ is set to 1, otherwise it is set to 0. But this is actually a decision-making process of a perceptron.
 On this basis, logistic regression needs to add a layerto find the relationship between the classification probability and the input variable, and judge the category through probability.
Recalling the linear regression model:$$h\left(x\right)=w^{T}x+b$$ Add a function ggon the basis of the linear model$g$，即$h\left(x\right)=g(w^{T}x+b)$ . This function is the sigmoid function, also called the logistic function$$g(z)=\frac{1}{1+e^{-z}}$$

It converts the result of a linear regression into a probability value. At this time $h\left(x\right)$ represents the probability of something happening, we can also record it as$p(Y=1|x)$
You can look at the image of the sigmoid function:

Summary: In this way, we get the Logistic model h ( x ) h(x)The expression of$h$$($$x$$)$$$h(x)=\frac{1}{1+e^{-w^{T}x+b}}$$
:$h(x_i)$ means that the sample$x_i$The probability that the label is 1 is $h(x)$

Loss function, loss minimization architecture

log loss as loss function

对数损失：$$l(y,\hat{y})=-\sum _{i=1}^m{y}_{i}lo{g}_2\hat{y}$$

when $When\; l$ takes the minimum value, the model$\hat{y}$The distribution closest to the true (theoretical) model $y_i$distributed

Loss Minimization Architecture

Because it is a binary classification problem, so

Feed it into the log loss function $lSo$
the loss minimization architecture of the two classifications is:

$$min-\sum [y_{i}log(h(x_i))+(1-y_i)log(1-h(x_i))]$$

Why doesn't the logistic regression loss function use the least squares method? Answer

---

This is the intersection:
$$-\sum \left[y_{i}log\right(h(x_i))+(1-y_i\left)log\right(1-h(x_i))]$$

---

From the perspective of probability theory and statistics, loss minimization architecture (cross entropy):
 In statistics, suppose we already have a set of samples (X, Y), in order to calculate the parameters that can generate this set of samples. Usually we will use the method of maximum likelihood estimation (a commonly used method of parameter estimation). If we use the maximum likelihood estimation, we also need a hypothetical estimation, here we are assuming $Y$ is subject tothe Bernoulli distribution. $$P(Y)=1|x)=p(x)$$$$P(Y)=0|x)=1-p\left(x\right)$$ due to$Y$ obeys the Bernoulli distribution, we can easily have the likelihood function:$$L=\prod [p(x_i{)}^{y_i}][1-p(x_i){]}^{(1-y_i)}$$ in order to solve it we can take the logarithm on both sides:
$$logL=\sum [y_{i}log(p(x_i))+(1-y_i)log(1-p(x_i))]$$

The maximum likelihood estimation is actually the probability model with the highest probability: you may not understand it, but in another way, $\prod p(h(x_i)|\theta )$ , the existing probability model$h\left(x\right)$ in the existing sampleθ \thetaUnder the condition of$\theta$ , the larger the calculated value, the more$h\left(x\right)$ is closest to the theoretical probability model

We generally like to take the minimum value of the formula, so convert the original formula to
$$min(-logL)=min-\sum [y_{i}log(p(x_i))+(1-y_i)log(1-p(x_i))]$$

Here $p\left(x_i\right)$ corresponds to$h(x_i)$

Looking at the loss minimization architecture from the perspective of information volume and entropy:

KL divergence: when $D_{KL}=0$ , P model = Q model, what we pursue is the Q model we built (that is,$h\left(x\right)$ ) is close to the real P model (here is the theoretical model of the sample,$p_{i}for$ the sample)

Yes, according to Gibbs inequality , cross entropy $\ge$ the entropy of the P system, so when we take the minimum valueof cross entropy, the Q model is closer to the real theoretical P model, and we know that the definition of information is$$f:=-lo{g}_{2}So\; bring\; x$$
into the original formula, the cross entropy is:$$-\sum _{i=1}^m{p}_{i}lo{g}_{2}x$$

This is where the log loss comes in: $$l(y,\hat{y})=-\sum _{i=1}^m{y}_{i}lo{g}_2\hat{y}$$When watching the teacher's class, he directly threw me a logarithmic loss, and then directly talked about cross entropy. Why logarithmic loss can be used as a loss function did not tell us. Through the above knowledge, readers should understand how this logarithmic loss came about. Keep reading, you will gain more.

Recommended video: [How does "cross-entropy" make a loss function? Package understanding "information amount", "bit", "entropy", "KL divergence", "cross entropy"]

Because it is a binary classification problem, so

So the cross entropy of the two categories is:

$$-\sum [y_{i}log(p(x_i))+(1-y_i)log(1-p(x_i))]$$

Here $p\left(x_i\right)$ corresponds to$h(x_i)$

Thinking from these two perspectives:

• The formulas derived from them are the same, can they all be called cross entropy?
  • No, they are just the same formula. When the maximum likelihood estimation function is derived, the $log$ , its base can be any positive number, and it has no unit, andthe amount of informationwill be$lo{g}_2$Written in the definition, and it has a unit, and the bit is its unit.

classification function

Maximum Probability Classification Function

in a $In\; the\; k$ -ary classification problem, given the prediction probability model$h$，即
$$h(x)=(h_1(x),h_2(x),...,h_k(x))$$ Among them,$h_i\left(x\right)$ is the probability that the sample belongs to the ith class. The maximum probability classification function
for model his: Max P robh ( x ) = argmaxhi ( x ) MaxProb_h(x)=argmax h_i(x)
$$MaxPro{b}_h(x)=argmax{h}_i\left(x\right)$$

:argmax, take the subscript of the maximum value.
This $h_i\left(x\right)$ can be understood as, the sample label is$h_i$The probability of the maximum $h_i$as the label of sample i

threshold classification function

In a 2-gram classification problem, let the labels take values ​​in {0,1}.
Given a prediction probability model h , ie h(x) represents the probability that the label of feature group x is 1. The threshold classification function
with t as the threshold for model h is:

That is, if the real label is 1, then when the function value of the prediction model (label is 1) is greater than a certain value, the prediction label is considered to be 1, and it is 0 anyway.

Optimization Algorithm for Logistic Regression

gradient descent

Theoretical part: Another article of mine: Search Algorithm - Study Notes

achine_learning.logistic_regression.lib.logistic_regression_gd
import numpy as np

def sigmoid(scores):
	return 1 / (1 + np.exp(-scores))

class LogisticRegression:
	def fit(self, X, y, eta=0.1, N = 1000): #eta η
		m, n = X.shape
		w = np.zeros((n,1))
		for t in range(N):
			h = sigmoid(X.dot(w)) #hw(X)
			g = 1.0 / m * X.T.dot(h - y) #梯度g
			w = w - eta * g #更新w
 		self.w = w
 	def predict_proba(self, X):
		return sigmoid(X.dot(self.w)) # 概率
		
	def predict(self, X):
		proba = self.predict_proba(X) # 概率
		return (proba >= 0.5).astype(np.int) 

Is cross entropy a convex function? It can be derived.

stochastic gradient descent

Theoretical part: Another article of mine: Search Algorithm - Study Notes

import numpy as np

def sigmoid(scores):
	return 1.0 / (1 + np.exp(-scorecs ))

class LogisticRegression:
	def fit(self, X, y, eta_0=10, eta_1=50, N=1000):
		m, n = X.shape
		w = np.zeros((n,1))
		self.w = w
		for t in range(N):
			i = np.random.randint(m) #随机选一个样本 x(i)
			x = X[i].reshape(1, -1) # 1*n 向量
			pred = sigmoid(x.dot(w)) # hw(x)
			g = x.T * (pred - y[i]) # 梯度
			w = w - eta_0 / (t + eta_1) * g # 更新w
			self.w += w
		self.w /= N # w均值

	def predict_proba(self, X):
		return sigmoid(X.dot(self.w))

	def predict(self, X):
		proba = self.predict_proba(X)
		return (proba >= 0.5).astype(np.int)

Mini-batch gradient descent

coordinate descent