最大熵模型-极大似然估计

最大熵模型（一）讲诉了最大熵原理以及最大熵模型定义，最大熵模型（二）讲诉了最大熵模型的学习及其公式推导，本篇讲诉最大熵模型的极大似然估计。

最大熵模型

P_{w} (y | x) = \frac{1}{Z_{w}} e x p (\sum_{i = 1}^{n} w_{i} f_{i} (x, y)))

$P_{w}(y|x) =\frac{1}{Z_{w}}exp(\sum_{i=1}^{n}w_if_i(x,y)))$
其中：

Z_{w} (x) = \sum_{y} e x p (\sum_{i = 1}^{n} w_{i} f_{i} (x, y))

$Z_{w}(x)=\sum_{y}exp(\sum_{i=1}^{n}w_if_i(x,y))$

极大似然估计

对于给定数据集 $T=\{(x_1,y_1),(x_2,y_2),\cdot\cdot\cdot,(x_N,y_N)\}$ ，其中 $x$ 的可能取值记为 $X = \{v_1,v2,\cdot\cdot\cdot,v_m\}$ ， $y$ 的可能取值记为 $Y = \{\gamma_1,\gamma_2,\cdot\cdot\cdot,\gamma_n\}$ 。用 $C(X=v_i,Y=\gamma_j)$ 表示样本 $(v_i,\gamma_j)$ 在数据集中出现的次数。

采用极大思然估计模型参数，似然函数为：

L (y_{1}, y_{2}, \cdot \cdot \cdot, y_{N} | x_{1}, x_{2}, \cdot \cdot \cdot, x_{N}) = \prod_{i = 1}^{N} p (x_{i} | y_{i}) = \prod_{X, Y} p (Y = γ_{j} | X = v_{i})^{C (X = v_{i}, Y = γ_{j})}

$L(y_1,y_2,\cdot\cdot\cdot,y_N|x_1,x_2,\cdot\cdot\cdot,x_N) = \prod_{i=1}^Np(x_i|y_i) =\prod_{X,Y}p(Y=\gamma_j|X=v_i)^{C(X=v_i,Y=\gamma_j)}$
两边同时开N次方，得：

\begin{aligned} L (y_{1}, y_{2}, \cdot \cdot \cdot, y_{N} | x_{1}, x_{2}, \cdot \cdot \cdot, x_{N})^{\frac{1}{N}} & = \prod_{X, Y} p (Y = γ_{i} | x = v_{j})^{\frac{C (X = v_{i}, Y = γ_{j})}{N}} \\ = \prod_{X, Y} p (Y = γ_{j} | X = v_{i})^{\tilde{p} (X = v_{i}, Y = γ_{j})} \end{aligned}

$\begin{align*} L(y_1,y_2,\cdot\cdot\cdot,y_N|x_1,x_2,\cdot\cdot\cdot,x_N)^{\frac{1}{N}} &=\prod_{X,Y}p(Y=\gamma_i|x=v_j)^{\frac{C(X=v_i,Y=\gamma_j)}{N}} \\ &=\prod_{X,Y}p(Y=\gamma_j|X=v_i)^{\widetilde{p}(X=v_i,Y=\gamma_j)} \end{align*}$

\tilde{p} (X = v_{i}, Y = γ_{j})

$\widetilde{p}(X=v_i,Y=\gamma_j)$ 表示数据集的经验概率分布。
对数似然为：

\begin{aligned} L_{\tilde{p}} (P_{w}) & = N l o g \prod_{X, Y} p (Y = γ_{j} | X = v_{i})^{\tilde{p} (X = v_{i}, Y = γ_{j})} \\ = N \sum_{X, Y} \tilde{p} (X = v_{i}, Y = γ_{j}) l o g p (Y = γ_{j} | X = v_{i}) \end{aligned}

$\begin{align*} L_\widetilde{p}(P_w) &= Nlog\prod_{X,Y}p(Y=\gamma_j|X=v_i)^{\widetilde{p}(X=v_i,Y=\gamma_j)} \\ &= N\sum_{X,Y}{\widetilde{p}(X=v_i,Y=\gamma_j)}logp(Y=\gamma_j|X=v_i) \\ \end{align*}$

L_{\tilde{p}} (P_{w}) \propto \sum_{X, Y} \tilde{p} (X = v_{i}, Y = γ_{j}) l o g p (Y = γ_{j} | X = v_{i})

$L_\widetilde{p}(P_w)\propto\sum_{X,Y}{\widetilde{p}(X=v_i,Y=\gamma_j)}logp(Y=\gamma_j|X=v_i)$
简记为：

L_{\tilde{p}} (P_{w}) = \sum_{x, y} \tilde{p} (x, y) l o g p (y | x)

$L_\widetilde{p}(P_w) =\sum_{x,y}{\widetilde{p}(x,y)}logp(y|x)$
当条件概率是最大熵模型时，有：

\begin{aligned} L_{\tilde{p}} (P_{w}) & = \sum_{x, y} \tilde{p} (x, y) (\sum_{i = 1}^{n} w_{i} f_{i} (x, y) - l o g Z_{w} (x)) \\ = \sum_{x, y} \tilde{p} (x, y) \sum_{i = 1}^{n} w_{i} f_{i} (x, y) - \sum_{x} \tilde{p} (x, y) l o g Z_{w} (x) \end{aligned}

$\begin{align*} L_\widetilde{p}(P_w) &= \sum_{x,y}{\widetilde{p}(x,y)}(\sum_{i=1}^{n}w_if_i(x,y)-logZ_{w}(x)) \\ &=\sum_{x,y}{\widetilde{p}(x,y)}\sum_{i=1}^{n}w_if_i(x,y)-\sum_{x}{\widetilde{p}(x,y)}logZ_{w}(x) \end{align*}$

最大熵模型的对偶函数：

\begin{aligned} L (P_{w}, w) & = - H (p_{w}) + \sum_{i = 1}^{n} w_{i} (E_{\tilde{p}} (f_{i}) - E_{p_{w}} (f_{i})) \\ = \sum_{x, y} \tilde{p} (x) p_{w} (y | x) l o g p_{w} (y | x) + \sum_{i = 1}^{n} w_{i} (\sum_{x, y} \tilde{p} (x, y) f_{i} (x, y) - \sum_{x, y} \tilde{p} (x) p_{w} (y | x) f_{i} (x, y)) \\ = \sum_{x, y} \tilde{p} (x) p_{w} (y | x) (\sum_{i = 1}^{n} w_{i} f_{i} (x, y) - l o g Z_{w} (x)) + \sum_{x, y} \tilde{p} (x, y) \sum_{i = 1}^{n} w_{i} f_{i} (x, y) - \sum_{x, y} \tilde{p} (x) p_{w} (y | x) \sum_{i = 1}^{n} w_{i} f_{i} (x, y) \\ = \sum_{x, y} \tilde{p} (x, y) \sum_{i = 1}^{n} w_{i} f_{i} (x, y) - \sum_{x, y} \tilde{p} (x, y) p_{w} (y | x) l o g Z_{w} (x) \\ = \sum_{x, y} \tilde{p} (x, y) \sum_{i = 1}^{n} w_{i} f_{i} (x, y) - \sum_{x} \tilde{p} (x, y) l o g Z_{w} (x) \end{aligned}

$\begin{align*} L(P_w,w) &= -H(p_w) + \sum_{i=1}^{n}w_i(E_{\widetilde{p}}(f_{i})-E_{p_w}(f_{i})) \\ &= \sum_{x,y}\widetilde{p}(x)p_w(y|x)logp_w(y|x) + \sum_{i=1}^{n}w_i(\sum_{x,y}\widetilde{p}(x,y)f_i(x,y)-\sum_{x,y}\widetilde{p}(x)p_w(y|x)f_i(x,y)) \\ &=\sum_{x,y}\widetilde{p}(x)p_w(y|x)(\sum_{i=1}^{n}w_if_i(x,y) - logZ_w(x))+\sum_{x,y}\widetilde{p}(x,y)\sum_{i=1}^{n}w_if_i(x,y) - \sum_{x,y}\widetilde{p}(x)p_w(y|x)\sum_{i=1}^{n}w_if_i(x,y)\\ &=\sum_{x,y}{\widetilde{p}(x,y)}\sum_{i=1}^{n}w_if_i(x,y)-\sum_{x,y}{\widetilde{p}(x,y)}p_w(y|x)logZ_{w}(x) \\ &=\sum_{x,y}{\widetilde{p}(x,y)}\sum_{i=1}^{n}w_if_i(x,y)-\sum_{x}{\widetilde{p}(x,y)}logZ_{w}(x) \end{align*}$

因此有：

L_{\tilde{p}} (P_{w}) = L (P_{w}, w)

$L_\widetilde{p}(P_w) = L(P_w,w)$

即最大熵模型学习中对偶函数极大化等价于最大熵模型的极大似然估计。

参考文献

李航《统计学习方法》
陈希孺《概率论与数理统计》

最大熵模型（三）

最大熵模型-极大似然估计