[Statistical Learning｜Book Reading] Chapter 6 Logistics Returning to China and Maximum Entropy Model p77-p88

train of thought

Logistic regression is a classic classification method for statistical learning. The maximum entropy is a criterion for probabilistic model learning, and it is extended to classification problems to obtain the maximum entropy model.

logistic regression model

Logistic distribution: Let X be a continuous random variable. X obeying logistic distribution means having the following distribution function and density function: $$F(x)=P(X\le x)=\frac{1}{1+e^{-(x-\mu )/c}}$$
$$f(x)=F^{{}^{\prime }}(x)=\frac{e^{-(x-\mu )/}}{c(1+e^{-(x-\mu )/c}{)}^2}$$

Logistics regression model: The binomial logistic regression model is the following conditional probability distribution: $$P(Y)=1|x)=\frac{exp(w\ast x+b)}{1+exp(w\ast x+b)}$$
$$P(Y)=0|x)=\frac{1}{1+exp(w\ast x+b)}$$

Parameter estimation of the model: The maximum likelihood estimation method can be used to estimate the model parameters, so as to obtain the logistic regression model. The log likelihood function is obtained as: $$L(w)=\sum _{i=1}^N[y_i(w\ast x_i)-log(1+exp(w\ast x_i)]$$ to$L\left(w\right)$ finds the maximum value to get the estimated value of w. In this way, the problem becomes an optimization function with the log-likelihood function as the objective function. The gradient descent method and Newton method are usually used in logistic regression learning.

The multinomial logistic regression model is the same.

maximum entropy model

Maximum Entropy Model Definition

The principle of maximum entropy can be expressed as selecting the model with the largest entropy in the set of models satisfying the constraints.

Maximum entropy model:

Defined in the conditional probability distribution P ( Y ∣ X ) P(Y|X)The conditional entropy on$P$$($$Y$$|$$X$$)$$$H(P)=-\sum _{x,y}\widehat{P(x)P(y|x)logP(y|x)}$$
Then the conditional entropy H ( P ) H(P) in the model set CThe model with the largest$H$$($$P$$)\; is\; called\; the\; maximum\; entropy\; model,\; and\; the\; logarithm\; in\; the\; formula\; is\; the\; natural\; logarithm.$

Learning of Maximum Entropy Models

The learning process of the maximum entropy model is the process of solving the maximum entropy model, and the learning of the maximum entropy model can be formalized as a constrained optimization problem.

Transform the original problem of constrained optimization into the dual problem of unconstrained optimization, and solve the original problem by seeking the dual problem.

So the obtained maximum entropy model is: $$P_w(y|x)=\frac{1}{Z_w(x)}exp(\sum _{i=1}^n{w}_i{f}_i(x,y))$$,其中$$Z_w\left(x\right)=\sum _{y}exp\left(\sum _{i=1}^n{w}_i{f}_i\right(x,y))$$,$Z_w\left(x\right)$ is a normative factor,$f_i(x,y)$ is the characteristic function;$w_i$is the weight of the feature, where w is the parameter vector in the maximum entropy model.

maximum likelihood estimation

The dual function maximization in the maximum entropy model learning is equivalent to the maximum likelihood estimation of the maximum entropy model. In this way, the learning problem of the maximum entropy model is transformed into the problem of solving the maximization of the logarithmic likelihood function or the maximization of the dual function.

Optimization Algorithms for Model Learning

The learning of the logistic regression model and the maximum entropy model boils down to the optimization problem with the likelihood function as the objective function, which is usually solved by an iterative algorithm. From the optimization point of view, the objective function at this time has good properties, it is a smooth convex function, so many optimization methods are applicable to ensure that the global optimal solution can be found. The commonly used methods are iterative scaling method, Gradient descent method, Newton method or quasi-Newton method. Newton's or quasi-Newton's methods generally converge faster.