[Eating the book together] "Machine Learning" Chapter 7 Bayesian Classifier

Chapter 7 Bayesian Classifiers

7.1 Bayesian decision theory

  For classification tasks, in the ideal situation where all relevant probabilities are known, Bayesian decision theory considers how to select the optimal category label based on these probabilities and misjudgment losses, assuming NNPossible class labels in$N$ , ie y = { c 1 , c 2 , . . . , c N } y = \{ {c_1},{c_2},...,{c_N}\}y={ c1​,c2​,...,cN​} ,$l_{ij}$is to mark a real as $c_j$of samples misclassified as $c_i$The resulting loss, based on the posterior probability $P(c_i|x)$ can obtain the sample$x$ is classified as$c_i$The resulting expected loss, that is, at sample x \bf{x}"Conditional risk" on$x$
$$R(c_i|\mathbf{x})=\sum _{j=1}^N{l}_{ij}P(c_j|x)$$
  Our task is to find a criterion$h:x\mapsto y$ to minimize the overall risk:
$$R(h)={\mathbb{E}}_{\mathbf{x}}\left[R\right(h\left(x\right)|x)]$$
  Obviously, for each sample$x$ , if$h$ can minimize the conditional risk$R\left(h\right(x)|x)$ , then the overall risk$R\left(h\right)$ will also be minimized, which leads to the Bayesian decision criterion: To minimize the overall risk, simply choose on each sample that conditional risk R ( c ∣ x ) R(c|$R(c|x)$ the smallest class label, that is,
$$h^{\ast }(\mathbf{x})=\underset{c\in y}{\mathrm{argmin}}R(c|x)$$
  At this time,$h^{\ast }$ is called the Bayesian optimal classifier, and the corresponding$R(h^{\ast })$is called Bayesian hazard,$1-R\left(h^{\ast }\right)$reflects the best performance that the classifier can achieve, that is, the theoretical upper limit of the model accuracy that can be produced by machine learning.

  It is not difficult to see that if you want to use the Bayesian criterion to minimize the decision risk, you must first obtain the posterior probability $P(c|x)$ , however, this is usually difficult to obtain directly in real tasks, so what machine learning needs to achieve is to estimate the posterior probability P ( c ∣ x )$P(c|x)$ , there are currently two strategies:

• Discriminant model: given $x$ , can be directly modeled by$P(c|x)$ to predict$c$
• Generative mode: First, the joint probability distribution $P(\mathbf{x},c)$ modeling, and then obtain$P(c|\mathbf{x})$

$$P(c|x)=\frac{P(x,c)}{P(\mathbf{x})}=\frac{P(c)P(\mathbf{x}|c)}{P\left(x\right)}$$

  where $P\left(c\right)$ is the class "prior" probability,$P(x|c)$ is the sample$x$ is relative to the class labelccThe class conditional probability (also called "likelihood") of$c$$P\left(x\right)$ is the "evidence" factor used for normalization. For a given sample$x$ , evidence factor$P\left(x\right)$ has nothing to do with the class label, so estimate$The\; problem\; of\; P(c|x)$ is transformed into howto$D$ to estimate prior$P\left(c\right)$ and likelihood$P(\mathbf{x}|c)$。

  Class prior probability $P\left(c\right)$ expresses the proportion of various samples in the sample space. According to the law of large numbers, when the training set contains sufficient independent and identically distributed samples,$P\left(c\right)$ can be estimated by the probability of occurrence of various samples.

  For class conditional probability $For\; P(x|c)$ , since it involves about$The\; joint\; probability\; of\; all\; attributes\; of\; x$ , directly based on the probability of sample occurrence, will encounter serious difficulties.

• Prior probability: A subjective probability estimate of an event or parameter before new information is observed.
• Posterior Probability: The reestimation of the probability of an event or parameter after new information has been observed.
• Likelihood: The probability of observing new information given an event or parameter is known.

7.2 Maximum Likelihood Estimation

  A common strategy for estimating class conditional probability is to assume that it has a certain form of probability distribution, and then estimate the parameters of the probability distribution based on training samples, assuming P ( x ∣ c ) $P(x|c)$ has a definite form and is parameterized by the vector$i_c$uniquely determined, then you need to use the training set $D$ to estimate the parameter$i_c$。

  The following is a brief introduction to the views of the two schools of statistics on parameter estimation:

• Frequentist school: The parameter is unknown, but the fixed value exists objectively, and the parameter value can be determined by optimizing the likelihood function and other criteria, which also forms statistical learning.
• Bayesian school: The parameter is an unobserved random variable, which itself has a distribution. It can be assumed that the parameter obeys a prior distribution, and then the posterior distribution of the parameter is calculated based on the observed data, which also forms Bayesian learning .

  Let $D_c$Denotes the training set DDccin$D$$A\; set\; of\; c$ samples, assuming that these samples are independent and identically distributed, the parameter$i_c$For dataset $D_c$Independent function
$$P(D_c|{\theta }_c)=\prod _{\mathbf{x}\in D_c}P(\mathbf{x}|{\theta }_c)$$
  Considering that the multiplication operation is likely to cause underflow (multiplication of small real values), the log likelihood is usually used, as shown below
$$LL({\theta }_c)=\log P(D_c|{\theta }_c)=\sum _{x\in D_c}\log P(x|{\theta }_c)$$
  so for$i_c$Carry out maximum likelihood estimation and find a way to maximize the likelihood $P(D_c|{\theta }_c)$ parameter value${\hat{i}}_c$,unlock
$${\hat{i}}_c=\underset{i_c}{\mathrm{argmax}}LL({\theta }_c)$$
  Let’s take an example of maximum likelihood

7.3 Naive Bayes Classifier

  The main difficulty in estimating the posterior probability based on the Bayesian formula is that the class conditional probability is the joint probability of all attributes, which is difficult to estimate directly from limited training samples; It will encounter the problem of combinatorial explosion, and it will encounter the problem of sample sparseness in the data (the more attributes, the more serious the problem).

  In order to avoid this obstacle, the Naive Bayesian classifier adopts the "attribute conditional independence assumption": for known categories, it is assumed that all attributes are independent of each other, that is, it is assumed that each attribute independently affects the classification result, as shown below , where $d$ is the number of attributes,$x_i$for $x$ presentiiThe value on the$i\; attribute.$
$$P(c|\mathbf{x})=\frac{P(c)P(\mathbf{x}|c)}{P(\mathbf{x})}=\frac{P(c)}{P\left(x\right)}\prod _{i=1}^{d}P(x_i|c)$$
  The Bayesian decision criterion is as follows
$$h_{nb}(\mathbf{x})=\underset{c\in y}{\mathrm{argmax}}P(c)\prod _{i=1}^{d}P(x_i|c)$$
  let$D_c$Denotes the training set DDccin$D$$For\; a\; set\; of\; c$ -type samples, if there are sufficient independent and identically distributed samples, the class prior probability$P(c)=\frac{|D_c|}{|D|}$。

  For discrete attributes, let $D_{c,x_i}$means $D_c$in the $The\; value\; of\; i$ attribute is$x_i$A set composed of samples, the conditional probability $P(x_i|c)$可估计为
$$P(x_i|c)=\frac{|D_{c,x_i}|}{\left|D_c\right|}$$
  For continuous attributes, the probability density function can be considered, assuming $p(x_i|c)\sim N(m_{c,i},p_{c,i}^2)$ , where$m_{c,i}$和$p_{c,i}^2$cc $Class\; c$ sample in$The\; mean\; and\; variance\; of\; the\; values\; on\; i$ attributes, then
$$p(x_i|c)=\frac{1}{\sqrt{2p.m}{p}_{c,i}}\exp (-\frac{{(x_i-m_{c,i})}^2}{2p_{c,i}^2})$$
  Let's take an example of the Naive Bayesian classifier

7.4 Semi-Naive Bayesian Classifiers

  The basic idea of ​​the semi-naive Bayesian classifier is to properly consider the interdependence information between some attributes, so that it does not need to calculate the complete joint probability, and it does not completely ignore the relatively strong attribute dependencies. "Independent dependence estimation" is the most commonly used strategy for semi-naive Bayesian classifiers, assuming that each attribute depends on at most one other attribute outside the category, that is, P ( c ∣ x )
$$P(c|\mathbf{x})\propto P(c)\prod _{i=1}^{d}P(x_i|c,p{a}_i)$$
  in that$p{a}_i$is attribute $x_i$Depends on the attribute, called $x_i$In order to determine the parent attribute of each attribute, the most direct way is to assume that all attributes depend on the same attribute, which is called "super parent", and then determine the super parent attribute through model selection methods such as cross-validation.

7.5 Bayesian Nets

  Bayesian networks use directed acyclic graphs to describe the dependencies between attributes, and use conditional probability tables to describe the joint probability distribution of attributes. Specifically, a Bayesian net $B$ by structure$G$ and parameters$\Theta$ consists of two parts, that is,$B=⟨G,\Theta ⟩$ . Network structure$G$ is a directed acyclic graph, each node of which corresponds to an attribute, if two attributes have a direct dependency, they are connected by an edge; the parameter Θ \$\Theta$ quantitatively describes this dependency, assuming the attributes$x_i$in GGThe set of parent nodes in$G$${Pi}_i$，则$\Theta$ containment condition summary table$i_{x_i|\pi {}_i}=P_B(x_i|{\pi }_i)$。

  Let’s illustrate the Bayesian network with an example. As shown in the figure below, "color" directly depends on "good melon" and "sweetness", while "root" directly depends on "sweetness". Further, the conditional probability table Get the quantitative dependence of "root" on "sweetness", such as $P$ (root=stiff|sweetness=high)=0.1 etc.

(1) Structure

  The Bayesian network structure effectively expresses the conditional independence between attributes. Given a set of parent nodes, the Bayesian network assumes that each attribute is independent of its non-descendant attributes, so B = ⟨ G , Θ ⟩ $B=⟨G,\Theta ⟩$ attributes$x_i,x_2,...,x_d$The joint probability distribution is defined as
$$P_B(x_1,x_2,...,x_d)=\prod _{i=1}^d{P}_B(x_i|{\pi }_i)=\prod _{i=1}^d{i}_{x_i|{\pi }_i}$$
  The figure below shows the typical dependencies among the three variables in the Bayesian network. In the "same parent" structure, given the parent node $x_1$value, then $x_3$with $x_4$Conditionally independent; in the "order" structure, given $The\; value\; of\; x$ , then$y$ and$z$ conditionally independent;$The\; V-$ shaped structure is also called "collision" structure, given the stator node$x_4$The value of $x_1$with $x_2$must not be independent, but if $x_4$The value of is completely unknown, then VVUnder the$V$$x_1$with $x_2$are independent of each other. Such independence is also called "marginal independence".

  To analyze conditional independence between variables in a directed graph, "directed separation" can be used:

• Find all $V-$ shaped structure,the VVA directed edge is added between the two parent nodes of the$V-\; shaped\; structure.$
• Change all directed edges to undirected edges.

(2) study

  The primary task of Bayesian network learning is to find the most "appropriate" Bayesian network according to the training data set. "Scoring search" is a common method to solve this problem. By defining a scoring function to evaluate the Bayesian network The degree of fit between the network and the training data, and then based on this scoring function to find the optimal structure of the Bayesian network.

  Commonly used scoring functions are usually based on information-theoretic criteria, which treat the learning problem as a data compression task, and the goal of learning is to find a model that can describe the training data with the shortest code length. For Bayesian network learning, the model is a Bayesian network. At the same time, each Bayesian network describes a probability distribution on the training data, and has its own set of encoding mechanisms to make those frequently occurring samples more accurate Short code, so we should choose the Bayesian network with the shortest comprehensive code length, which is the "minimum description length" criterion.

  Given training set D = { x 1 , x 2 , . . . , xm } D = \{{x_1},{x_2},...,{x_m}\}D={ x1​,x2​,...,xm​} , Bayesian network$B=⟨G,\Theta ⟩$ in$The\; scoring\; function\; on\; D$ can be written as follows, where$|B|$ is the number of parameters of the Bayesian network,$f\left(\theta \right)$ means describing each parameterθ \thetaThe number of encoding bits required for$\theta \; .$
$$s(B|D)=f(i)\left|B\right|-LL(B|D)$$
  The first item in the formula is the calculation code Bayesian networkBBThe number of coding bits required for$B$ , the second item is to calculate BBThe probability distribution PBcorresponding to$B\; P\_B$$P_B$to $D$ how well described. Therefore, the learning task is transformed into an optimization task, which is to find a Bayesian network$B$ makes the scoring function$s(B|D)$ is the smallest.

  若$f\left(i\right)=1$ , that is, each parameter uses$1$ encoding bit description, then get$AIC$ scoring function
$$AIC(B|D)=\left|B\right|-LL(B|D)$$
  若$f\left(i\right)=\frac{1}{2}\log m$ , that is, each parameter uses$\frac{1}{2}\log m$ encoding bit description, then get$BIC$ scoring function
$$BIC(B|D)=\frac{\log m}{2}\left|B\right|-LL(B|D)$$
  f$f\left(i\right)=0$ , that is, the length of encoding the network is not calculated, then the scoring function degenerates into negative log likelihood, and correspondingly, the learning task degenerates into maximum likelihood estimation.

(3) Inference

  Inference refers to the process of inferring the variable to be queried through the observed value of the known variable. The ideal is to accurately calculate the posterior probability directly according to the joint probability distribution defined by the Bayesian network, but this is an NP $NP-$ hard problems, in practical applications, the approximate inference of the Bayesian network is often completed by Gibbs sampling, which is a random sampling method, as shown below.

7.6 EM algorithm

  Real life often encounters "incomplete" training samples, for example, the roots of watermelons have fallen off, and it is impossible to tell whether it is "curled up" or "stiff". The scientific name for unobserved variables is "hidden variables". Let $X$ represents the observed variable set,$Z$ represents the hidden variable set,$\Theta$ denotes the model parameters. For$\Theta$ is used for maximum likelihood estimation, the log likelihood should be maximized
$$LL(\Theta |\mathbf{X},\mathbf{Z})=\ln P(\mathbf{X},Z|\Theta )$$
  However, since$Z$ is a hidden variable, and the above formula cannot be solved directly. At this time, it can be solved by$Z$ computes the expectation to maximize the log "marginal likelihood" of the observed data
$$LL(\Theta |\mathbf{X})=\ln P(\mathbf{X}|\Theta )=\ln \sum {}_{\mathbf{Z}}P(\mathbf{X},Z|\Theta )$$
  $The\; EM$ algorithm is a commonly used tool for estimating parameter hidden variables. It is an iterative method, and its basic idea is: if the parameter$\Theta$ is known, then the optimal latent variable Z \bf{Z}can be inferred based on the training data$The\; value\; of\; Z$ ($E$ step); On the contrary, if$If\; the\; value\; of\; Z$ is known, it is convenient for the parameter$\Theta do$ maximum likelihood estimation ($Step\; M$ ).

  Therefore, with the initial value ${Th}^0$ as the starting point, the following steps can be performed iteratively until convergence:

• Based on ${Th}^t$ infers the hidden variable$The\; expectation\; of\; Z$ , denoted as${\mathbf{Z}}^{\mathbf{t}}$；
• Based on the observed variable $\mathbf{X}$和$Z^t$ versus parameter$\Theta$ is used for maximum likelihood estimation, recorded as${Th}^{t+1}$；

  If we are not taking $The\; expectation\; of\; Z$ , but based on${Th}^t$ calculates the hidden variableZ \bf{Z}The probability distribution P of$Z$$P(\mathbf{Z}|\mathbf{X},{Th}^t)$I'm$The\; two\; steps\; of\; the\; EM$ algorithm are:

• $Step\; E$ : With the current parameter${Th}^t$ infers the hidden variable distribution$P(Z|X,{Th}^t)$, and calculate the log-likelihood$LL(\Theta |\mathbf{X},Z)$About$Z$ 's expectations

$$Q(\Theta |{\Theta }^t)={\mathbb{E}}_{Z|X,{Th}^t}LL(\Theta |\mathbf{X},\mathbf{Z})$$

• $M-$ step: find the parameters to maximize the expected likelihood, ie

$${Th}^{t+1}=\underset{Th}{\mathrm{argmax}}Q(\Theta |{\Theta }^t)$$