版权声明：本文为博主原创文章，未经博主允许不得转载。 https://blog.csdn.net/Hemk340200600/article/details/86246410
在自然语言处理NLP领域中，词袋模型（bag of words,BOW）是经典的模型之一。它考虑所有的词汇都装在一个袋子中，任何一篇文档中的单词都可以用袋子中的词汇来描述。如果有10万个词汇，那么每篇文档都可以表示为一个10万维的向量。得到文档向量之后，我们通过可以通过描述向量之间的差异来寻找文档之间的关联。
稀疏性(Sparseness)：对于大词袋，尤其是包含了生僻字的词袋，文档稀疏性不可避免。
一词多义：一词多义是一种常见的现象，但是BOW模型只统计单词出现的次数，而忽略了他们之间的区别；
多词一义：在不同或相同文档中，可能存在多种单词存在同一个意思。
LSA也称为LSI(latent semantic index)。在PCA降维中，我们利用SVD做矩阵分解，将样本降到低纬度的空间。在LSA中，我们同样也应用SVD来解决相似的问题，将文档从稀疏的n维空间映射到一个低维（k维）的向量空间，我们称之为隐含语义空间(Latent Semantic Space).【机器学习算法推导】奇异值分解SVD与主成分分析PCA
（1）分析文档集合，建立词汇-文本矩阵A。
（2）对词汇-文本矩阵进行奇异值分解。
（3）使用降维后的矩阵构建潜在语义空间，将原始文本映射到k维空间。
将一篇文档映射到低维空间，意味着有一些词汇在降维后在同一个点上，也就是说，这些点刻画着同义词。
降维可去除部分噪声，使特征更鲁棒；
充分利用冗余数据；
无监督/完全自动化；
与语言无关；
没有刻画词汇出现次数的概率模型；
无法解决多义词的问题；
SVD的优化目标基于L-2 norm 或者是 Frobenius Norm的，这相当于隐含了对数据的高斯噪声假设。而term出现的次数是非负的，这明显不符合Gaussian假设;
对于count vectors 而言，欧式距离表达是不合适的（重建时会产生负数）；
特征向量的方向没有对应的物理解释；
SVD的计算复杂度很高，而且当有新的文档来到时，若要更新模型需重新训练；
对于一个正常人来说，我们是如何构思一篇文章的呢？一般来说我们会先确定我们要写的主题，比如说我正在写的这篇文章是机器学习相关的，可能会涉及到计算机，概率论，机器学习方面的术语，假设线性代数占20%，概率论主题占30%，机器学习主题占50%。而在概率论中，可能会联想到期望，方差，概率等词汇，线性代数可能会联想到矩阵，特征值等词汇，机器学习方面可能会出现模型，权重，空间等词汇。
p
(
d
m
)
p(d_m)
p ( d m )
d
m
d_m
d m
p
(
z
k
∣
d
m
)
p(z_k|d_m)
p ( z k ∣ d m )
z
k
z_k
z k
p
(
w
n
∣
z
k
)
p(w_n|z_k)
p ( w n ∣ z k )
w
n
w_n
w n
p
(
d
m
,
z
k
,
w
n
)
=
P
(
d
m
)
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
)
p(d_m,z_k,w_n)=P(d_m)P(z_k|d_m)P(w_n|z_k)
p ( d m , z k , w n ) = P ( d m ) P ( z k ∣ d m ) P ( w n ∣ z k )
w
⃗
=
(
w
1
,
w
2
,
.
.
,
w
N
)
\vec w=(w_1,w_2,..,w_N)
w
= ( w 1 , w 2 , . . , w N )
P
(
w
⃗
∣
d
m
)
=
∏
n
=
1
N
P
(
w
n
∣
d
m
)
P(\vec w|d_m)=\prod\limits_{n=1}^NP(w_n|d_m)
P ( w
∣ d m ) = n = 1 ∏ N P ( w n ∣ d m )
d
m
d_m
d m
w
n
w_n
w n
P
(
d
m
,
w
n
)
=
P
(
d
m
)
P
(
w
n
∣
d
m
)
=
P
(
d
m
)
∑
k
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
,
d
m
)
=
P
(
d
m
)
∑
k
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
)
\begin{aligned} P(d_m,w_n) &=P(d_m)P(w_n|d_m) \\ &=P(d_m)\sum_kP(z_k|d_m)P(w_n|z_k,d_m) \\ &=P(d_m)\sum_kP(z_k|d_m)P(w_n|z_k) \end{aligned}
P ( d m , w n ) = P ( d m ) P ( w n ∣ d m ) = P ( d m ) k ∑ P ( z k ∣ d m ) P ( w n ∣ z k , d m ) = P ( d m ) k ∑ P ( z k ∣ d m ) P ( w n ∣ z k )
P
(
w
n
∣
z
k
)
P(w_n|z_k)
P ( w n ∣ z k )
P
(
z
k
∣
d
m
)
P(z_k|d_m)
P ( z k ∣ d m )
L
(
θ
)
=
ln
∏
m
=
1
M
∏
n
=
1
N
P
(
d
m
,
w
n
)
n
(
d
m
,
w
n
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
ln
P
(
d
m
,
w
n
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
ln
(
P
(
d
m
)
∑
k
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
)
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
ln
∑
k
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
)
+
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
ln
P
(
d
m
)
\begin{aligned} L(\theta) &=\ln\prod_{m=1}^M\prod_{n=1}^NP(d_m,w_n)^{n(d_m,w_n)} \\ &=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\ln P(d_m,w_n) \\ &=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\ln (P(d_m)\sum_kP(z_k|d_m)P(w_n|z_k)) \\ &=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\ln \sum_kP(z_k|d_m)P(w_n|z_k)+ \sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\ln P(d_m) \end{aligned}
L ( θ ) = ln m = 1 ∏ M n = 1 ∏ N P ( d m , w n ) n ( d m , w n ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) ln P ( d m , w n ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) ln ( P ( d m ) k ∑ P ( z k ∣ d m ) P ( w n ∣ z k ) ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) ln k ∑ P ( z k ∣ d m ) P ( w n ∣ z k ) + m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) ln P ( d m )
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
ln
P
(
d
m
)
\sum\limits_{m=1}^M\sum\limits_{n=1}^Nn(d_m,w_n)\ln P(d_m)
m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) ln P ( d m )
L
(
θ
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
ln
∑
k
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
ln
∑
k
P
(
w
n
,
z
k
∣
d
m
)
\begin{aligned} L(\theta)&=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\ln \sum_kP(z_k|d_m)P(w_n|z_k)\\ &=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\ln \sum_kP(w_n,z_k|d_m) \end{aligned}
L ( θ ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) ln k ∑ P ( z k ∣ d m ) P ( w n ∣ z k ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) ln k ∑ P ( w n , z k ∣ d m )
L
(
θ
,
θ
i
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
∑
k
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
ln
P
(
w
m
,
z
k
∣
d
m
)
L(\theta,\theta_i)=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\sum_kP(z_k|w_n,d_m;\theta_i)\ln P(w_m,z_k|d_m)
L ( θ , θ i ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) k ∑ P ( z k ∣ w n , d m ; θ i ) ln P ( w m , z k ∣ d m )
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
=
P
(
z
k
,
w
n
,
d
m
;
θ
i
)
P
(
w
n
,
d
m
;
θ
i
)
=
P
(
d
m
;
θ
i
)
P
(
z
k
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
k
;
θ
i
)
P
(
d
m
;
θ
i
)
P
(
w
n
∣
d
m
;
θ
i
)
=
P
(
z
k
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
k
;
θ
i
)
P
(
w
n
∣
d
m
;
θ
i
)
=
P
(
z
k
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
k
;
θ
i
)
∑
j
P
(
z
j
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
j
;
θ
i
)
\begin{aligned} P(z_k|w_n,d_m;\theta_i) &=\frac{P(z_k,w_n,d_m;\theta_i)}{P(w_n,d_m;\theta_i)} \\ &=\frac{P(d_m;\theta_i)P(z_k|d_m;\theta_i)P(w_n|z_k;\theta_i)}{P(d_m;\theta_i)P(w_n|d_m;\theta_i)} \\ &=\frac{P(z_k|d_m;\theta_i)P(w_n|z_k;\theta_i)}{P(w_n|d_m;\theta_i)} \\ &=\frac{P(z_k|d_m;\theta_i)P(w_n|z_k;\theta_i)}{\sum_jP(z_j|d_m;\theta_i)P(w_n|z_j;\theta_i)} \\ \end{aligned}
P ( z k ∣ w n , d m ; θ i ) = P ( w n , d m ; θ i ) P ( z k , w n , d m ; θ i ) = P ( d m ; θ i ) P ( w n ∣ d m ; θ i ) P ( d m ; θ i ) P ( z k ∣ d m ; θ i ) P ( w n ∣ z k ; θ i ) = P ( w n ∣ d m ; θ i ) P ( z k ∣ d m ; θ i ) P ( w n ∣ z k ; θ i ) = ∑ j P ( z j ∣ d m ; θ i ) P ( w n ∣ z j ; θ i ) P ( z k ∣ d m ; θ i ) P ( w n ∣ z k ; θ i )
P
(
w
m
,
z
k
∣
d
m
)
=
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
)
P(w_m,z_k|d_m)=P(z_k|d_m)P(w_n|z_k)
P ( w m , z k ∣ d m ) = P ( z k ∣ d m ) P ( w n ∣ z k )
L
(
θ
,
θ
i
)
L(\theta,\theta_i)
L ( θ , θ i )
L
(
θ
,
θ
i
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
∑
k
P
(
z
k
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
k
;
θ
i
)
∑
j
P
(
z
j
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
j
;
θ
i
)
ln
P
(
z
k
∣
d
m
)
P
(
w
n
∣
z
k
)
=
∑
m
=
1
M
∑
n
=
1
N
n
(
d
m
,
w
n
)
∑
k
P
(
z
k
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
k
;
θ
i
)
∑
j
P
(
z
j
∣
d
m
;
θ
i
)
P
(
w
n
∣
z
j
;
θ
i
)
(
ln
P
(
z
k
∣
d
m
)
+
ln
P
(
w
n
∣
z
k
)
)
\begin{aligned} L(\theta,\theta_i) &=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\sum_k\frac{P(z_k|d_m;\theta_i)P(w_n|z_k;\theta_i)}{\sum_jP(z_j|d_m;\theta_i)P(w_n|z_j;\theta_i)}\ln P(z_k|d_m)P(w_n|z_k) \\ &=\sum_{m=1}^M\sum_{n=1}^Nn(d_m,w_n)\sum_k\frac{P(z_k|d_m;\theta_i)P(w_n|z_k;\theta_i)}{\sum_jP(z_j|d_m;\theta_i)P(w_n|z_j;\theta_i)}(\ln P(z_k|d_m)+\ln P(w_n|z_k)) \end{aligned}
L ( θ , θ i ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) k ∑ ∑ j P ( z j ∣ d m ; θ i ) P ( w n ∣ z j ; θ i ) P ( z k ∣ d m ; θ i ) P ( w n ∣ z k ; θ i ) ln P ( z k ∣ d m ) P ( w n ∣ z k ) = m = 1 ∑ M n = 1 ∑ N n ( d m , w n ) k ∑ ∑ j P ( z j ∣ d m ; θ i ) P ( w n ∣ z j ; θ i ) P ( z k ∣ d m ; θ i ) P ( w n ∣ z k ; θ i ) ( ln P ( z k ∣ d m ) + ln P ( w n ∣ z k ) )
θ
i
+
1
=
arg
max
θ
L
(
θ
,
θ
i
)
w
i
t
h
∑
n
P
(
w
n
∣
z
k
)
=
1
,
∑
k
P
(
z
k
∣
d
m
)
=
1
\theta_{i+1}=\arg \max_\theta L(\theta,\theta_i) \\ with \quad \sum_nP(w_n|z_k)=1,\sum_kP(z_k|d_m)=1
θ i + 1 = arg θ max L ( θ , θ i ) w i t h n ∑ P ( w n ∣ z k ) = 1 , k ∑ P ( z k ∣ d m ) = 1
L
(
θ
)
=
L
(
θ
,
θ
i
)
+
∑
k
α
k
(
1
−
∑
n
P
(
w
n
∣
z
k
)
)
+
∑
m
β
m
(
1
−
∑
k
P
(
z
k
∣
d
m
)
)
\mathcal{L}(\theta)=L(\theta,\theta_i)+\sum_k\alpha_k(1-\sum_nP(w_n|z_k))+\sum_m\beta_m(1-\sum_kP(z_k|d_m))
L ( θ ) = L ( θ , θ i ) + k ∑ α k ( 1 − n ∑ P ( w n ∣ z k ) ) + m ∑ β m ( 1 − k ∑ P ( z k ∣ d m ) )
P
(
w
n
∣
z
k
)
P(w_n|z_k)
P ( w n ∣ z k )
P
(
z
k
∣
d
m
)
P(z_k|d_m)
P ( z k ∣ d m )
∂
L
∂
P
(
w
n
∣
z
k
)
=
∑
m
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
P
(
w
n
∣
z
k
)
−
α
k
=
0
,
1
≤
n
≤
N
,
1
≤
k
≤
K
\frac{\partial \mathcal{L}}{\partial P(w_n|z_k)}=\frac{\sum_mn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}{P(w_n|z_k)} -\alpha_k=0,1 \le n \le N,1 \le k \le K
∂ P ( w n ∣ z k ) ∂ L = P ( w n ∣ z k ) ∑ m n ( d m , w n ) P ( z k ∣ w n , d m ; θ i ) − α k = 0 , 1 ≤ n ≤ N , 1 ≤ k ≤ K
∂
L
∂
P
(
z
k
∣
d
m
)
=
∑
n
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
P
(
z
k
∣
d
m
)
−
β
m
=
0
,
1
≤
m
≤
M
,
1
≤
k
≤
K
\frac{\partial \mathcal{L}}{\partial P(z_k|d_m)}=\frac{\sum_nn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}{P(z_k|d_m)} -\beta_m=0,1 \le m \le M,1 \le k \le K
∂ P ( z k ∣ d m ) ∂ L = P ( z k ∣ d m ) ∑ n n ( d m , w n ) P ( z k ∣ w n , d m ; θ i ) − β m = 0 , 1 ≤ m ≤ M , 1 ≤ k ≤ K
P
(
w
n
∣
z
k
)
=
∑
m
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
α
k
P(w_n|z_k)=\frac{\sum_mn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}{\alpha_k}
P ( w n ∣ z k ) = α k ∑ m n ( d m , w n ) P ( z k ∣ w n , d m ; θ i )
P
(
z
k
∣
d
m
)
=
∑
n
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
β
m
P(z_k|d_m)=\frac{\sum_nn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}{\beta_m}
P ( z k ∣ d m ) = β m ∑ n n ( d m , w n ) P ( z k ∣ w n , d m ; θ i )
∑
n
P
(
w
n
∣
z
k
)
=
1
,
∑
k
P
(
z
k
∣
d
m
)
=
1
\sum_nP(w_n|z_k)=1,\sum_kP(z_k|d_m)=1
∑ n P ( w n ∣ z k ) = 1 , ∑ k P ( z k ∣ d m ) = 1
P
(
w
n
∣
z
k
;
θ
i
+
1
)
=
∑
m
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
∑
n
∑
m
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
P(w_n|z_k;\theta_{i+1})=\frac{\sum_mn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}{\sum_n\sum_mn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}
P ( w n ∣ z k ; θ i + 1 ) = ∑ n ∑ m n ( d m , w n ) P ( z k ∣ w n , d m ; θ i ) ∑ m n ( d m , w n ) P ( z k ∣ w n , d m ; θ i )
P
(
z
k
∣
d
m
;
θ
i
+
1
)
=
∑
n
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
∑
k
∑
n
n
(
d
m
,
w
n
)
P
(
z
k
∣
w
n
,
d
m
;
θ
i
)
P(z_k|d_m;\theta_{i+1})=\frac{\sum_nn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}{\sum_k\sum_nn(d_m,w_n)P(z_k|w_n,d_m;\theta_i)}
P ( z k ∣ d m ; θ i + 1 ) = ∑ k ∑ n n ( d m , w n ) P ( z k ∣ w n , d m ; θ i ) ∑ n n ( d m , w n ) P ( z k ∣ w n , d m ; θ i )
P
(
w
n
∣
z
k
)
P(w_n|z_k)
P ( w n ∣ z k )
P
(
z
k
∣
d
m
)
P(z_k|d_m)
P ( z k ∣ d m )