13.1 试推导出式(13.5)~(13.8)

式13.5

首先，高斯混合模型的公式如下：

p M (x) \sum i = 1 k α i = \sum i = 1 k α i \cdot p (x ∣ μ i, Σ i), = 1 (1) (2)

$\begin{align} p_{\mathcal M}(\mathbf x)&=\sum_{i=1}^k \alpha_i \cdot p(\mathbf x \mid \mu_i,\Sigma_i),\tag{1} \\ \sum_{i=1}^k \alpha_i &= 1 \tag{2}\end{align}$
则根据贝叶斯定理，未标记样本

xj $x_j$ 属于各高斯混合成分

Θi $\Theta_i$ 的概率为：

γ j i = p (Θ = i ∣ x j) = p ( Θ = i ) \cdot p ( x j ∣ Θ = i ) p M ( x j ) = α i \cdot p ( x j ∣ μ i , Σ i ) \sum i = 1 k α i \cdot p ( x j ∣ μ i , Σ i ), (3)

$\gamma_{ji}=p(\Theta=i \mid \mathbf x_j)={{p(\Theta=i)\cdot p(\mathbf x_j \mid \Theta=i )}\over {p_{\mathcal M}(\mathbf x_j)}}={{\alpha_i \cdot p(\mathbf x_j \mid \mu_i,\Sigma_i)} \over {\displaystyle\sum_{i=1}^k \alpha_i \cdot p(\mathbf x_j \mid \mu_i,\Sigma_i)}},\tag{3}$ 则式

(13.5) $(13.5)$ 得证。

式13.6~13.7

根据书上公式 $(13.4)$ :

L L (D l \cup D u) = \sum (x j, y j) \in D l ln (\sum i = 1 k α i \cdot p (x j ∣ μ i, Σ i) \cdot p (y j ∣ Θ = i, x j)) + \sum x j \in D u ln (\sum i = 1 k α i \cdot p (x j ∣ μ i, Σ i)) (4)

$LL(D_l \cup D_u)=\sum_{(\mathbf x_j,y_j) \in D_l} \ln \left(\sum_{i=1}^k \alpha_i \cdot p(\mathbf x_j \mid \mu_i,\Sigma_i) \cdot p(y_j \mid \Theta=i,\mathbf x_j) \right)+\sum_{\mathbf x_j \in D_u} \ln \left(\sum_{i=1}^k \alpha_i \cdot p(\mathbf x_j \mid \mu_i,\Sigma_i)\right) \tag{4}$
由于假定每一个类别对应一个高斯混合成分，则

p(yj∣Θ=i,xj)=1 $p(y_j \mid \Theta=i,\mathbf x_j)=1$ 当且仅当

yj=i $y_j=i$ ，否则

p(yj∣Θ=i,xj)=0 $p(y_j \mid \Theta=i,\mathbf x_j)=0$ ，则上式

(4) $(4)$ 可以化简为：

L L (D l \cup D u) = \sum (x j, y j) \in D l \land y j = i ln α i \cdot p (x j ∣ μ i, Σ i) + \sum x j \in D u ln (\sum i = 1 k α i \cdot p (x j ∣ μ i, Σ i)) (5)

$LL(D_l \cup D_u)=\sum_{(\mathbf x_j,y_j) \in D_l\land y_j=i} \ln \quad \alpha_i \cdot p(\mathbf x_j \mid \mu_i,\Sigma_i) +\sum_{\mathbf x_j \in D_u} \ln \left(\sum_{i=1}^k \alpha_i \cdot p(\mathbf x_j \mid \mu_i,\Sigma_i)\right) \tag{5}$
接下来回顾一下高斯分布的表达式：

p (x ∣ μ i, Σ i) = 1 ( 2 π ) n 2 | Σ i | 1 2 exp {- 1 2 (x - μ i) ⊤ Σ - 1 i (x - μ i)} (6)

$p(\mathbf x \mid \mu_i,\Sigma_i) ={1\over (2\pi)^{n \over 2}\vert \Sigma_i \vert^{1 \over 2}} \exp \left\{-{1\over 2} (\mathbf x-\mu_i)^\top \Sigma_i^{-1} (\mathbf x-\mu_i) \right\}\tag{6}$
其对参数的偏导为：

\partial p ( x ∣ μ i , Σ i ) \partial μ i \partial p ( x ∣ μ i , Σ i ) \partial Σ i = p (x ∣ μ i, Σ i) \cdot Σ - 1 i \cdot (μ i - x) = p (x ∣ μ i, Σ i) \cdot Σ - 2 i \cdot ((x - μ i) (x - μ i) ⊤ - Σ i) (7) (8)

$\begin{align} {{\partial p(\mathbf x\mid {\mu}_i,\Sigma_i)}\over{\partial \mu_i}} &= p(\mathbf x\mid {\mu}_i,\Sigma_i)\cdot \Sigma_i^{-1} \cdot (\mu_i-\mathbf x) \tag{7} \\ {{\partial p(\mathbf x\mid {\mu}_i,\Sigma_i)}\over{\partial \Sigma_i}} &= p(\mathbf x\mid {\mu}_i,\Sigma_i) \cdot \Sigma_i^{-2} \cdot \left((\mathbf x-\mu_i)(\mathbf x-\mu_i)^\top-\Sigma_i\right) \tag{8} \end{align}$
以

(5) $(5)$ 对

αi $\alpha_i$ 求偏导，将

(3),(7) $(3),(7)$ 带入得：

\partial L L ( D l \cup D u ) \partial μ i = \sum (x j, y j) \in D l \land y j = i α i \cdot p ( x j ∣ μ i , Σ i ) α i \cdot p ( x j ∣ μ i , Σ i ) \cdot Σ - 1 i \cdot (μ i - x j) + \sum x j \in D u α i \cdot p ( x j ∣ μ i , Σ i ) \sum i = 1 k α i \cdot p ( x j ∣ μ i , Σ i ) \cdot Σ - 1 i \cdot (μ i - x j) = Σ - 1 ⎛ ⎝ \sum (x j, y j) \in D l \land y j = i (μ i - x j) + \sum x j \in D u γ j i (μ i - x j) ⎞ ⎠ (9)

$\begin{align} {{\partial LL(D_l \cup D_u)}\over{\partial \mu_i}} &=\sum_{(\mathbf x_j,y_j) \in D_l\land y_j=i}{\alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)\over \alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)}\cdot \Sigma_i^{-1} \cdot (\mu_i-\mathbf x_j) +\sum_{\mathbf x_j \in D_u}{\alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)\over \displaystyle \sum_{i=1}^k \alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)}\cdot \Sigma_i^{-1} \cdot (\mu_i-\mathbf x_j) \\ &=\Sigma^{-1}\left(\sum_{(\mathbf x_j,y_j) \in D_l \land y_j=i}(\mu_i-\mathbf x_j) + \sum_{\mathbf x_j \in D_u}\gamma_{ji}(\mu_i-\mathbf x_j) \right) \tag{9} \end{align}$
令

(9)=0 $(9)=0$ 可解得：

μ i = 1 \sum i = 1 k γ j i + l i ⎛ ⎝ \sum (x j, y j) \in D l \land y j = i x j + \sum x j \in D u γ j i x j ⎞ ⎠ (10)

$\mu_i = {1\over {\displaystyle\sum_{i=1}^k \gamma_{ji}+l_i}}\left(\sum_{(\mathbf x_j,y_j) \in D_l \land y_j=i}\mathbf x_j + \sum_{\mathbf x_j \in D_u}\gamma_{ji}\mathbf x_j \right) \tag{10}$ 则

(13.6) $(13.6)$ 得证。
同样地，以

(5) $(5)$ 对

Σi $\Sigma_i$ 求偏导，将

(3),(8) $(3),(8)$ 带入得：

\partial L L ( D l \cup D u ) \partial Σ i = \sum (x j, y j) \in D l \land y j = i α i \cdot p ( x j ∣ μ i , Σ i ) α i \cdot p ( x j ∣ μ i , Σ i ) \cdot Σ - 2 i \cdot ((x - μ i) (x - μ i) ⊤ - Σ i) + \sum x j \in D u α i \cdot p ( x j ∣ μ i , Σ i ) \sum i = 1 k α i \cdot p ( x j ∣ μ i , Σ i ) \cdot Σ - 2 i \cdot ((x - μ i) (x - μ i) ⊤ - Σ i) = Σ - 2 ⎛ ⎝ \sum (x j, y j) \in D l \land y j = i ((x - μ i) (x - μ i) ⊤ - Σ i) + \sum x j \in D u γ j i ((x - μ i) (x - μ i) ⊤ - Σ i) ⎞ ⎠ (11)

$\begin{align} {{\partial LL(D_l \cup D_u)}\over{\partial \Sigma_i}} &=\sum_{(\mathbf x_j,y_j) \in D_l\land y_j=i}{\alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)\over \alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)}\cdot \Sigma_i^{-2} \cdot \left((\mathbf x-\mu_i)(\mathbf x-\mu_i)^\top-\Sigma_i\right) \\ &\quad +\sum_{\mathbf x_j \in D_u}{\alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)\over \displaystyle \sum_{i=1}^k \alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)}\cdot \Sigma_i^{-2} \cdot \left((\mathbf x-\mu_i)(\mathbf x-\mu_i)^\top-\Sigma_i\right) \\ &=\Sigma^{-2}\left(\sum_{(\mathbf x_j,y_j) \in D_l \land y_j=i}\left((\mathbf x-\mu_i)(\mathbf x-\mu_i)^\top-\Sigma_i\right) + \sum_{\mathbf x_j \in D_u}\gamma_{ji}\left((\mathbf x-\mu_i)(\mathbf x-\mu_i)^\top-\Sigma_i\right) \right) \tag{11} \end{align}$
令

(11)=0 $(11)=0$ 可解得：

Σ i = 1 \sum i = 1 k γ j i + l i ⎛ ⎝ \sum (x j, y j) \in D l \land y j = i (x - μ i) (x - μ i) ⊤ + \sum x j \in D u γ j i (x - μ i) (x - μ i) ⊤ ⎞ ⎠ (12)

$\Sigma_i = {1\over {\displaystyle\sum_{i=1}^k \gamma_{ji}+l_i}}\left(\sum_{(\mathbf x_j,y_j) \in D_l \land y_j=i}(\mathbf x-\mu_i)(\mathbf x-\mu_i)^\top + \sum_{\mathbf x_j \in D_u}\gamma_{ji}(\mathbf x-\mu_i)(\mathbf x-\mu_i)^\top \right) \tag{12}$ 则

13.7 $13.7$ 得证。

式13.8

对于混合系数 $\alpha_i$ ，除了要最大化 $LL(D_l\cup D_u)$ ，还应满足隐含条件： $\alpha_i \ge0,\sum_{i=1}^N\alpha_i=1$ ，因此考虑对 $LL(D_l\cup D_u)$ 使用拉格朗日乘子法，变为优化

L L (D l \cup D u) + λ (\sum i = 1 N α i - 1) (13)

$LL(D_l\cup D_u)+\lambda \left(\sum_{i=1}^N\alpha_i-1\right) \tag{13}$
将

(5) $(5)$ 带入

(13) $(13)$ ，并令

(13) $(13)$ 对

αi $\alpha_i$ 的导数为0，得到：

0 = \sum (x j, y j) \in D l \land y j = i p ( x j ∣ μ i , Σ i ) α i \cdot p ( x j ∣ μ i , Σ i ) + \sum x j \in D u p ( x j ∣ μ i , Σ i ) \sum i = 1 k α i \cdot p ( x j ∣ μ i , Σ i ) + λ (14)

$0=\sum_{(\mathbf x_j,y_j) \in D_l\land y_j=i}{p(\mathbf x_j\mid {\mu}_i,\Sigma_i)\over \alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)} +\sum_{\mathbf x_j \in D_u}{p(\mathbf x_j\mid {\mu}_i,\Sigma_i)\over \displaystyle \sum_{i=1}^k \alpha_i \cdot p(\mathbf x_j\mid {\mu}_i,\Sigma_i)}+\lambda \tag{14}$
对

(14) $(14)$ 两边同乘

αi $\alpha_i$ 得：

0 = \sum (x j, y j) \in D l \land y j = i 1 + \sum x j \in D u γ j i + α i λ (15)

$0=\sum_{(\mathbf x_j,y_j) \in D_l\land y_j=i}1 +\sum_{\mathbf x_j \in D_u}\gamma_{ji}+\alpha_i\lambda \tag{15}$
令

(15) $(15)$ 对所有混合高斯成分求和：

0 = \sum i = 1 k \sum (x j, y j) \in D l \land y j = i 1 + \sum i = 1 k \sum x j \in D u γ j i + \sum i = 1 k α i λ = l + u + λ (16)

$\begin{align} 0&=\sum_{i=1}^k\sum_{(\mathbf x_j,y_j) \in D_l\land y_j=i}1 +\sum_{i=1}^k\sum_{\mathbf x_j \in D_u}\gamma_{ji}+\sum_{i=1}^k\alpha_i\lambda \\ &=l+u+\lambda \tag{16} \end{align}$
解得

λ=−m $\lambda=-m$ ，将其带入

(15) $(15)$ 可得：

α i = 1 m ⎛ ⎝ \sum x j \in D u γ j i + l i ⎞ ⎠ (17)

$\alpha_i={1 \over m}\left(\sum_{\mathbf x_j \in D_u}\gamma_{ji}+l_i \right) \tag{17}$ 则

(13.8) $(13.8)$ 得证。

机器学习-周志华-个人练习13.1

13.1 试推导出式(13.5)~(13.8)

式13.5

式13.6~13.7

式13.8

猜你喜欢