"Statistical Learning Methods" - Maximum Likelihood Estimation of Naive Bayesian Parameters

References:

• "Statistical Learning Methods" Li Hang
• https://www.zhihu.com/question/33959624

Step1. Likelihood function

In the Naive Bayesian model, the parameters we need to determine through the training set are $i_k=P(and=c_k)$ 和 $m_{jl}=P(x^{(j)}=a_{jl}|y=c_k)$

Likelihood function:
$$\begin{array}{rl}L(\theta ,m)& =\prod _{i=1}^{N}P(x_i,y_i)\\ & =\prod _{i=1}^{N}P({and}_i)P(x_i|y_i)(multiplication\; formula)\\ & =\prod _{i=1}^N\left(P\right({and}_i\left)\prod _{j=1}^{n}P\right(x_i^{(j)}|y_i))(conditional\; independence\; assumption)\\ & =\prod _{i=1}^N\prod _{k=1}^K\left(P\right(and=c_k\left)\prod _{j=1}^{n}P\right(x_i^{\left(j\right)}|y_i=c_k){)}^{I(y_i=c_k)}\\ & =\prod _{i=1}^N\prod _{k=1}^K(i_k\prod _{j=1}^n\prod _{l=1}^{L_j}{P}^{I(x_i^{\left(j\right)}=a_{jl})}(x^{(j)}=a_{jl}|y_i=c_k){)}^{I(y_i=c_k)}\\ & =\prod _{i=1}^N\prod _{k=1}^K(i_k\prod _{j=1}^n\prod _{l=1}^{L_j}{m}_{jlk\_}^{I(x_i^{(j)}=a_{jl})}{)}^{I(y_i=c_k)}\end{array}$$
Among them, $N$ is the number of samples,$n$ isXXDimension of$X$$L_j$for $X^{\left(j\right)}$ The number of possible values,$K$为YYThe number of possible values ​​for$Y.$

Form:
$$\begin{array}{rl}l(i,m& =\sum _{i=1}^N\sum _{k=1}^{K}I(y_i=c_k)(\log i_k+\sum _{j=1}^n\sum _{l=1}^{L_j}I(x_i^{(j)}=a_{jl})\log m_{jl})\end{array}$$

Step2 If $i_k$

Use the Lagrange multiplier method to introduce constraints $\sum _{k=1}^K{i}_k=1$ ,得:
$$\begin{array}{r}F(\theta ,m,l)=\sum _{i=1}^N\sum _{k=1}^{K}I(y_i=c_k)(\log i_k+\sum _{j=1}^n\sum _{l=1}^{L_j}I(x_i^{(j)}=a_{jl})\log m_{jl})+l(\sum _{k=1}^K{i}_k-1)\end{array}$$

to $F$ Find the partial derivative and let the partial derivative be$0$ ,i:
$$\begin{array}{rl}{i}_k& =-\frac{\sum _{i=1}^{N}I(y_i=c_k)}{l}\\ \sum _{k=1}^K{i}_k& =-\frac{N}{l}=1\end{array}$$
Among them, $N_k$For the sample $Y=c_k$quantity. Combining the above two equations, we get:
$$\begin{array}{r}{i}_k=\frac{\sum _{i=1}^{N}I(y_i=c_k)}{N}\end{array}$$

Step3. Find $m_{lk}$

Use the Lagrange multiplier method to introduce constraints $\sum _{l=1}^{L_j}{m}_{lk}=1$ ,得:
$$\begin{array}{r}F(\theta ,m,l)=\sum _{i=1}^N\sum _{k=1}^{K}I(y_i=c_k)(\log i_k+\sum _{j=1}^n\sum _{l=1}^{L_j}I(x_i^{(j)}=a_{jl})\log m_{jl})+l(\sum _{l=1}^{L_j}{m}_{lk}-1)\end{array}$$

to $F$ Find the partial derivative and let the partial derivative be$0$ ，得：
$$\begin{array}{rl}{m}_{jl}& =-\frac{\sum _{i=1}^{N}I(y_i=c_k,x_i^{(j)}=a_{jl})}{l}\\ \sum _{l=1}^{L_j}{m}_{lk}& =-\frac{\sum _{i=1}^{N}I(y_i=c_k)}{l}=1\end{array}$$
联立上面两个方程，得：
$$\begin{array}{r}{m}_{jl}=\frac{\sum _{i=1}^{N}I(y_i=c_k,x_i^{\left(j\right)}=a_{jl})}{\sum _{i=1}^{N}I(y_i=c_k)}\end{array}$$