Word2Vec模型总结

1. Huffman树的构造
   解析：给定n个权值作为n个叶子节点，构造一棵二叉树，若它的带权路径长度达到最小，则称这样的二叉树为最优二叉树，也称Huffman树。数的带权路径长度规定为所有叶子节点的带权路径长度之和。Huffman树构造，如下所示：
   （1）将$\{w_1,w_2,...,w_3\}$看成是有n颗树的森林；
   （2）在森林中选出两个根节点的权值最小的树合并，作为一棵新树的左、右子树，且新树的根节点权值为其左、右子树根节点权值之和；
   （3）从森林中删除选取的两颗树，并将新树加入森林；
   （4）重复（2）（3）步，直到森林中只剩一棵树为止，该树即为所求的Huffman树。
   说明：利用Huffman树设计的二进制前缀编码，称为Huffman编码，它既能满足前缀编码条件，又能保证报文编码总长最短。

2. 基于Hierarchical Softmax的模型（CBOW模型）
   解析：
   

   
   其中参数的物理意义，如下所示：
   （1）${{\bf{X}}_w} = \sum\limits_{i = 1}^{2c} {{\bf{v}}\left( {Context{{\left( w \right)}_i}} \right) \in {\rm{R}^m}}$
   （2）$d_j^w$表示路径${p^w}$中第$j$结点对应的编码（根结点不对应编码）
   （3）$\theta _j^w$表示路径${p^w}$中第$j$非叶子结点对应的向量
   （4）${p^w}$表示从根结点出发到达$w$对应叶子结点的路径。
   （5）${l^w}$表示路径${p^w}$中包含结点的个数。
   Hierarchical Softmax基本思想，如下所示：
   $$p\left( {w|Context\left( w \right)} \right) = \prod\limits_{j = 2}^{{l^w}} {p\left( {d_j^w|{{\bf{x}}_w},\theta _{j - 1}^w} \right)}$$
   $$p\left( {d_j^w|{{\bf{x}}_w},\theta _{j - 1}^w} \right) = {\left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]^{1 - d_j^w}} \cdot {\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]^{d_j^w}}$$
   对于word2vec中基于Hierarchical Softmax的CBOW模型，优化的目标函数，如下所示：
   $$L = \sum\limits_{w \in C} {\log p\left( {w|Context\left( w \right)} \right)}$$
   这样得到对数似然函数，如下所示：
   $$L = \sum\limits_{w \in C} {\log \prod\limits_{j = 2}^{{l^w}} {\left\{ {{{\left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]}^{1 - d_j^w}}\cdot{{\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]}^{d_j^w}}} \right\}} } \\ = \sum\limits_{w \in C} {\sum\limits_{j = 2}^{{l^w}} {\left\{ {\left( {1 - d_j^w} \right)\cdot\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right] + d_j^w\cdot\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]} \right\}} }$$
   将花括号中的内容简记为$L\left( {w,j} \right)$，如下所示：
   $$L\left( {w,j} \right) = \left( {1 - d_j^w} \right) \cdot \log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right] + d_j^w \cdot \log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]$$
   使用随机梯度上升法对$\theta _{j - 1}^w$求偏导，如下所示：
   $$\begin{array}{l} \frac{{\partial L\left( {w,j} \right)}}{{\partial \theta _{j - 1}^w}} = \frac{\partial }{{\partial \theta _{j - 1}^w}}\left\{ {\left( {1 - d_j^w} \right)\cdot\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right] + d_j^w\cdot\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]} \right\} \\ = \left( {1 - d_j^w} \right)\cdot\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]{{\bf{x}}_w} - d_j^w\cdot\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right){{\bf{x}}_w} \\ {\rm{ = }}\left\{ {\left( {1 - d_j^w} \right)\cdot\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right] - d_j^w\cdot\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right\}{{\bf{x}}_w} \\ {\rm{ = }}\left[ {1 - d_j^w{\rm{ - }}\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]{{\bf{x}}_w} \\ \end{array}$$
   $\theta_{j-1}^w$的更新方程，如下所示：
   $$\theta _{j - 1}^w: = \theta _{j - 1}^w + \eta \left[ {1 - d_j^w{\rm{ - }}\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]{{\bf{x}}_w}$$
   使用随机梯度上升法对${{\bf{x}}_w}$求偏导，如下所示：
   $$\frac{{\partial L\left( {w,j} \right)}}{{\partial {{\bf{x}}_w}}} = \left[ {1 - d_j^w{\rm{ - }}\sigma \left( {{\bf{x}}_w^{\rm{T}}\theta _{j - 1}^w} \right)} \right]\theta _{j - 1}^w$$
   对于词典中每个词的词向量${\bf{v}}\left( {\tilde w} \right)$更新方程，如下所示：
   $${\bf{v}}\left( {\tilde w} \right): = {\bf{v}}\left( {\tilde w} \right) + \eta \sum\limits_{j = 2}^{{l^w}} {\frac{{\partial L\left( {w,j} \right)}}{{\partial {{\bf{x}}_w}}}} ,\tilde w \in Context\left( w \right)$$

3. 基于Hierarchical Softmax的模型（Skip-Gram模型）
   解析：
   

   
   其中，${\bf{v}}\left( w \right) \in {{\rm{R}}^m}$表示当前样本的中心词$w$的词向量。
   对于word2vec中基于Hierarchical Softmax的Skip-Gram模型，优化的目标函数，如下所示：
   $$L = \sum\limits_{w \in C} {\log p\left( {Context\left( w \right)|w} \right)}$$
   Skip-Gram模型中条件概率函数$p\left( {Context\left( w \right)|w} \right)$，如下所示：
   $$p\left( {Context\left( w \right)|w} \right){\rm{ = }}\prod\limits_{u \in Context\left( w \right)} {p\left( {u|w} \right)}$$
   $$p\left( {u|w} \right) = \prod\limits_{j = 2}^{{l^u}} {p\left( {d_j^u|{\bf{v}}\left( w \right),\theta _{j - 1}^u} \right)}$$
   $$p\left( {d_j^u|{\bf{v}}\left( w \right),\theta _{j - 1}^u} \right) = {\left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right]^{1 - d_j^u}} \cdot {\left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right]^{d_j^u}}$$
   这样得到对数似然函数，如下所示：
   $$\begin{array}{l} L = \sum\limits_{w \in C} {\log \prod\limits_{u \in Context\left( w \right)} {\prod\limits_{j = 2}^{{l^u}} {\left\{ {{{\left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right]}^{1 - d_j^u}} \cdot {{\left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right]}^{d_j^u}}} \right\}} } } \\ = \sum\limits_{w \in C} {\sum\limits_{u \in Context\left( w \right)} {\sum\limits_{i = 2}^{{l^u}} {\left\{ {\left( {1 - d_j^u} \right) \cdot \log \left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right] + d_j^u \cdot \log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right]} \right\}} } } \\ \end{array}$$
   将花括号中的内容简记为$L\left( {w,u,j} \right)$，如下所示：
   $$L\left( {w,u,j} \right) = \left( {1 - d_j^u} \right) \cdot \log \left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right] + d_j^u \cdot \log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}\theta _{j - 1}^u} \right)} \right]$$

4. 基于Negative Sampling的模型（CBOW模型）
   Negative Sampling不再使用Huffman树，而是使用随机负采样，能大幅度提高性能。假定已经选好$w$的负样本子集$NEG\left( w \right) \ne \emptyset$，定义词$\tilde w$的标签（正样本为1，负样本为0），如下所示：
   

   $${L^w}\left( {\tilde w} \right) = \left\{ \begin{array}{l} 1,\tilde w = w \\ 0,\tilde w \ne w \\ \end{array} \right.$$
   对于给定的正样本$\left( {Context\left( w \right),w} \right)$，最大化$g\left( w \right)$，如下所示：
   $$g\left( w \right) = \prod\limits_{u \in \left\{ w \right\} \cup NEG\left( w \right)} {p\left( {u|Context\left( w \right)} \right)}$$
   $$p\left( {u|Context\left( w \right)} \right) = \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]^{L^{w}\left( u \right)} \cdot {\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]^{\left[ {1 - {L^{w}\left( u \right)}} \right]}}$$
   其中，${{\bf{x}}_w}$表示$Context\left( w \right)$中各词的词向量之和，${\theta ^u} \in {{\rm{R}}^m}$表示词$u$对应的一个辅助向量，为待训练的参数。简化$g\left( w \right)$方程，如下所示：
   $$g\left( w \right) = \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^w}} \right)\prod\limits_{u \in NEG\left( w \right)} {\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]}$$
   其中，$\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^w}} \right)$表示当上下文为$Context\left( w \right)$时，预测中心词为$w$的概率，同样$\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right),u \in NEG\left( w \right)$表示当上下文为$Context\left( w \right)$时，预测中心词为$u$的概率。
   对于给定的语料库$C$，目标函数如下所示：
   $$\begin{array}{*{20}{c}} {L = \log G = \log \prod\limits_{w \in C} {g\left( w \right) = \sum\limits_{w \in C} {\log g\left( w \right)} } } \\ { = \sum\limits_{w \in C} {\log } {\prod _{u \in \left\{ w \right\} \cup NEG\left( w \right)}}\left\{ {{{\left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]}^{{L^w}\left( u \right)}}\cdot{{\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]}^{1 - {L^w}\left( u \right)}}} \right\}} \\ { = \sum\limits_{w \in C} {{\sum _{u \in \left\{ w \right\} \cup NEG\left( w \right)}}\left\{ {{L^w}\left( u \right)\cdot\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right]\cdot\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]} \right\}} } \\ { = \sum\limits_{w \in C} {\left\{ {\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^w}} \right)} \right] + {\sum _{u \in NEG\left( w \right)}}\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]} \right\}} } \\ { = \sum\limits_{w \in C} {\left\{ {\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^w}} \right)} \right] + {\sum _{u \in NEG\left( w \right)}}\log \left[ {\sigma \left( { - {\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]} \right\}} } \\ \end{array}$$
   记$L\left( {w,u} \right) = {L^w}\left( u \right)\cdot\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right]\cdot\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]$，使用随机梯度上升法对${\theta ^u}$求偏导，如下所示：
   $$\begin{array}{l} \frac{{\partial L\left( {w,u} \right)}}{{\partial {\theta ^u}}}{\rm{ = }}\frac{\partial }{{\partial {\theta ^u}}}\left\{ {{L^w}\left( u \right)\cdot\log \left[ {\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right]\cdot\log \left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]} \right\} \\ = {L^w}\left( u \right)\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]{{\bf{x}}_w} - \left[ {1 - {L^w}\left( u \right)} \right]\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right){{\bf{x}}_w} \\ = \left\{ {{L^w}\left( u \right)\left[ {1 - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right] - \left[ {1 - {L^w}\left( u \right)} \right]\sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right\}{{\bf{x}}_w} \\ = \left[ {{L^w}\left( u \right) - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]{{\bf{x}}_w} \\ \end{array}$$
   参数$\theta ^u$的更新方程，如下所示：
   $${\theta ^u}: = {\theta ^u} + \eta \left[ {{L^w}\left( u \right) - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]{{\bf{x}}_w}$$
   使用随机梯度上升法对${{\bf{x}}_w}$求偏导，如下所示：
   $$\frac{{\partial L\left( {w,u} \right)}}{{\partial {{\bf{x}}_w}}}{\rm{ = }}\left[ {{L^w}\left( u \right) - \sigma \left( {{\bf{x}}_w^{\rm{T}}{\theta ^u}} \right)} \right]{\theta ^u}$$
   参数${\bf{v}}\left( {\tilde w} \right),\tilde w \in Context\left( w \right)$的更新方程，如下所示：
   $${\bf{v}}\left( {\tilde w} \right): = {\bf{v}}\left( {\tilde w} \right) + \eta \sum\limits_{u \in \left\{ w \right\} \cup NEG\left( w \right)} {\frac{{\partial L\left( {w,u} \right)}}{{\partial {{\bf{x}}_w}}}} ,\tilde w \in Context\left( w \right)$$
   

5. 基于Negative Sampling的模型（Skip-Gram模型）
   对于给定的语料库$C$，目标函数如下所示：
   

   $$G = \prod\limits_{w \in C} {\prod\limits_{u \in Context\left( w \right)} {g\left( u \right)} }$$
   $$g\left( u \right) = \prod\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {p\left( {z|w} \right)}$$
   $$p\left( {z|w} \right) = {\left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]^{{L^u}\left( z \right)}} \cdot {\left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]^{1 - {L^u}\left( z \right)}}$$
   $$\begin{array}{l} L = \log G = \log \prod\limits_{w \in C} {\prod\limits_{_{u \in Context\left( w \right)}} {g\left( u \right)} } = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\log g\left( u \right)} } \\ = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\log \prod\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {p\left( {z|w} \right)} } } \\ = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\sum\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {\log \left\{ {{{\left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]}^{{L^u}\left( z \right)}} \cdot {{\left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]}^{1 - {L^u}\left( {\rm{z}} \right)}}} \right\}} } } \\ = \sum\limits_{w \in C} {\sum\limits_{_{u \in Context\left( w \right)}} {\sum\limits_{z \in \left\{ u \right\} \cup NEG\left\{ u \right\}} {\left\{ {{L^u}\left( z \right) \cdot \log \left[ {\sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right] + \left[ {1 - {L^u}\left( {\rm{z}} \right)} \right] \cdot \log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( w \right)}^{\rm{T}}}{\theta ^z}} \right)} \right]} \right\}} } } \\ \end{array}$$
   对每一个样本$\left( {w,Context\left( w \right)} \right)$，需要针对$Context\left( w \right)$中的每一个词进行负采样，但是word2vec源码中只是针对$w$进行了$\left| {Context\left( w \right)} \right|$次负采样。它本质上用的还是CBOW模型，只是将原来通过求和累加做整体用的上下文$Context\left( w \right)$拆成一个一个来考虑。对于给定的语料库$C$，目标函数如下所示：
   $${g\left( w \right) = {\prod _{\tilde w \in Context\left( w \right)}}{\prod _{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)}}p\left( {u|\tilde w} \right)}$$
   $${p\left( {u|\tilde w} \right) = {{\left[ {\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]}^{{L^w}\left( u \right)}}\cdot{{\left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]}^{1 - {L^w}\left( u \right)}}}$$
   $$\begin{array}{l} L = \log G = \log \prod\limits_{w \in C} {g\left( w \right)} = \sum\limits_{w \in C} {\log g\left( w \right)} \\ = \sum\limits_{w \in C} {\log \prod\limits_{\tilde w \in Context\left( w \right)} {\prod\limits_{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)} {\left\{ {{{\left[ {\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]}^{{L^w}\left( u \right)}} \cdot {{\left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]}^{1 - {L^w}\left( u \right)}}} \right\}} } } \\ = \sum\limits_{w \in C} {\log \sum\limits_{\tilde w \in Context\left( w \right)} {\sum\limits_{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)} {\left\{ {{L^w}\left( u \right) \cdot \log \left[ {\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right] \cdot \log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]} \right\}} } } \\ \end{array}$$
   记$L\left( {w,\tilde w,u} \right) = {L^w}\left( u \right) \cdot \log \left[ {\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right] \cdot \log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]$。使用随机梯度上升法，对${\theta ^u}$求偏导，如下所示：
   $$\begin{array}{l} \frac{{\partial L\left( {w,\tilde w,u} \right)}}{{\partial {\theta ^u}}} = \frac{{\partial L}}{{\partial {\theta ^u}}}\left\{ {{L^w}\left( u \right)\cdot\log \left[ {\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right] + \left[ {1 - {L^w}\left( u \right)} \right]\cdot\log \left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]} \right\} \\ = {L^w}\left( u \right)\left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]{\bf{v}}\left( {\tilde w} \right) - \left[ {1 - {L^w}\left( u \right)} \right]\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right){\bf{v}}\left( {\tilde w} \right) \\ = \left\{ {{L^w}\left( u \right)\left[ {1 - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right] - \left[ {1 - {L^w}\left( u \right)} \right]\sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right\}{\bf{v}}\left( {\tilde w} \right) \\ = \left[ {{L^w}\left( u \right) - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]{\bf{v}}\left( {\tilde w} \right) \\ \end{array}$$
   ${\theta ^u}$的更新方程，如下所示：
   $${\theta ^u}: = {\theta ^u} + \eta \left[ {{L^w}\left( u \right) - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]{\bf{v}}\left( {\tilde w} \right)$$
   使用随机梯度上升法，对${\bf{v}}\left( {\tilde w} \right)$求偏导，如下所示：
   $$\frac{{\partial L\left( {w,\tilde w,u} \right)}}{{\partial {\bf{v}}\left( {\tilde w} \right)}} = \left[ {{L^w}\left( u \right) - \sigma \left( {{\bf{v}}{{\left( {\tilde w} \right)}^{\rm{T}}}{\theta ^u}} \right)} \right]{\theta ^u}$$
   参数${{\bf{v}}\left( {\tilde w} \right)}$的更新，如下所示：
   $${\bf{v}}\left( {\tilde w} \right): = {\bf{v}}\left( {\tilde w} \right) + \eta \sum\limits_{u \in \left\{ w \right\} \cup NE{G^{\tilde w}}\left( w \right)} {\frac{{\partial L\left( {w,\tilde w,u} \right)}}{{\partial {\bf{v}}\left( {\tilde w} \right)}}}$$
   其中，$NE{G^{\tilde w}}\left( w \right)$表示处理词$\tilde w$时生成的负样本子集。

6. Negative Sampling算法
   （1）带权采样原理
   设词典$D$中的每一个词$w$对应一个线段$l\left( {w} \right)$，长度如下所示：
   $$len\left( w \right) = \frac{{{\rm{counter}}\left( w \right)}}{{\sum\limits_{u \in D} {{\rm{counter}}\left( u \right)} }}$$
   这里${{\rm{counter}}\left( \cdot \right)}$表示一个词在语料$C$中出现的次数。现在将这些线段首尾相连地拼接在一起，形成一个长度为1的单位线段。如果随机地往这个单位线段上打点，那么其中长度越长的线段（对应高频词）被打中的概率就越大。
   （2）word2vec负采样
   记$l_{0}=0$，${l_k} = \sum\limits_{j = 1}^k {len\left( {{w_j}} \right)} ,k = 1,2, \cdots ,N$，这里$w_{j}$表示词典$D$中第$j$个词，则以$\left\{ {{l_j}} \right\}_{j = 0}^N$为剖分结点可得到区间$\left[ {0,1} \right]$上的一个非等距剖分，${I_i} = ({l_{i - 1}},{l_i}],i = 1,2, \cdots ,N$为其$N$个剖分区间。进一步引入区间$\left [{0,1}\right]$上的一个等距离剖分，剖分结点为$\left\{ {{m_j}} \right\}_{j = 0}^M$，其中$M \gg N$，具体示意图如下所示：
   
   将内部剖分结点$\left\{ {{m_j}} \right\}_{j = 1}^{M-1}$投影到非等距剖分上，则可建立$\left\{ {{m_j}} \right\}_{j = 1}^{M-1}$与区间$\left\{ {{I_j}} \right\}_{j = 1}^{N}$（或$\left\{ {{w_j}} \right\}_{j = 1}^N$）的映射关系，如下所示：
   $${\rm{Table}}\left( i \right) = {w_k},{m_i} \in {I_k},i = 1,2, \cdots ,M - 1$$
   根据映射每次生成一个$\left[ {1,M - 1} \right]$间的随机整数$r$，${\rm{Table}}\left( r \right)$就是一个样本。当对$w_i$进行负采样时，如果采样为$w_i$，那么就跳过去。

参考文献：
[1] word2vec中的数学原理详解