Understanding word2vec

This article is my personal understanding of word2vec. Most of it is translated from Dr. Xin Rong's paper. At the same time, I refer to many blogs on the Internet, all of which are listed in References.
Note: In the mathematical formulas in the text, lowercase without bold represents scalar values, lowercase with bold represents vectors, and uppercase with bold represents matrices.

1. Introduction to word2vec

Word2Vec is a word vector (Word Embedding) calculation tool open sourced by Google in 2013, which is used to solve the distribution coding problem of words. It is widely used in natural language processing by learning semantic knowledge in an unsupervised manner from large amounts of text predictions .

2. What is word embedding (word embedding)

We know that text is a kind of unstructured data information, such as Chinese characters are a kind of square characters, which cannot be directly calculated. Therefore, it is necessary to convert this unstructured information into structured information, so as to realize calculation for text information. Simply put, it is to convert text into mathematical vectors.

There are many ways to express this, and there are three common ones:

• One-hot encoding | one-hot representation
• integer encoding
• word embedding | word embedding

The so-called one-hot encoding is to open a vector with the same size as the vocabulary, and a position of the vector represents a word. For example, assuming that there are only four words in the vocabulary, I, Love, China, and Country, then a possible encoding is :

Me: [1 0 0 0]
Love: [0 1 0 0] China
: [0 0 1 0]
Country: [0 0 0 1]

But in reality, tens of thousands of different words are likely to appear in the text, and the vector will be very long at this time. More than 99% of them are 0, leading to the so-called curse of dimensionality.

Integer encoding is simpler, which is to use an integer to represent a word, such as 1 for me, 2 for love, and so on.

The disadvantages of integer encoding are as follows:

• Inability to express relationship between words
• Integer encoding can be challenging for model interpretation.

Word embedding is a method of text representation. He can express the text through a low-dimensional vector , which is not as long as one-hot. And words with similar semantics will be relatively similar in the vector space . Word2vec is a method based on statistical methods to obtain word embeddings, which was proposed by Mikolov of Google in 2013.

The Word2vec algorithm has 2 training modes (2 network structures, CBOW and Skip-Gram):

1. CBOW (Continuous Bag-of-Word Model): Use the context of the word to predict the current word
2. Skip-gram: the current word predicts its context

First of all, word2vec is essentially a neural network, its input is the one-hot vector of the word, the output is the one-hot vector of the word to be predicted (not the so-called word embedding), and the real word embedding representation of the word is Implied in the parameters of the network.

Write down your own understanding below, most of which are explained in the paper word2vec Parameter Learning Explained by Dr. Xin Rong.

3. CBOW

CBOW uses context to predict the current word, that is, the input is a word with multiple contexts, and the output is a word to be predicted. First, for simplicity, we set the input context word to one.

3. 1 One-word context

At this time, the model inputs a word and predicts a word. The following figure shows the structure of the network:

In fact, the network structure is very simple. There is only one hidden layer in the figure, and the hidden layer has no activation function . The output layer has a Softmax function , that is, the probability distribution of the output predicted words, and the ones with high probability are predicted. up. We assume that the length of the word list is $V$ , that is, there are a total of$V$ words. The number of nodes in the hidden layer is$N$ , the number of nodes in the output layer is VVas in the input layer$V$ , because the probability of each word is to be output, and the true value is the one-hot of the word to be predicted, then the entire network must be trained to output one-hot as close as possible to the word to be predicted.

There is a fully connected network between the input layer and the hidden layer. Since there is no activation function, the input-output relationship is very simple:
$$\mathbf{h}={\mathbf{W}}^T\mathbf{x}={\mathbf{v}}_I^T$$
where $x$ is a$V\times A\; column\; vector\; of\; 1$ . $W$ is a$V\times N$ matrix, then$h$ is an$N\times A\; column\; vector\; of\; 1$ . Since$x$ has only one element that is$1$ , assuming the$The\; k$ dimension is 1, and the others are$0$ , then the essence of this operation is to put$W$ 'skkthAfter transposing$k$$\mathbf{h}$，即 ${\mathbf{v}}_I^T$, subscript $I$ stands for input.

Similarly, a size of N × VN \times V is passed between the hidden layer and the output layer$N\times V$ 's matrix${\mathbf{W}}^{\prime }$ full connection ($W^{\prime }$和$W$ has nothing to do with it! ), let its output be$u$ , then the output$j$个电影的的：
$$u_j={{\mathbf{v}}_j^{\prime }}^{T}h$$
where${v_j^{\prime }}^T$ represents the matrix${{\mathbf{W}}^{\prime }}^T$ 's$row\; j$ (or, matrix$W^{\text{'}}$ The$column\; j$ ). Finally, use Softmax to obtain the posterior probability of the word (the sum of all probabilities here is equal to$1$ 了）：
$$p\left(w_j\mid w_I\right)=y_j=\frac{\exp \left(u_j\right)}{{\sum }_{j^{\prime }=1}^V\exp \left(u_{j^{\prime }}\right)}$$
$p\left(w_j\mid w_I\right)$ represents the given input word, predicting output$The\; probability\; of\; j$ words. Substitute$u_j$，有，
$$p\left(w_j|w_I\right)=\frac{\exp \left(v_j^{\prime }{\mathbf{v}}_I\right)}{{\sum }_{j^{\prime }=1}^V\exp \left({\mathbf{v}}_{j^{\prime }}^{\prime }{}^T{\mathbf{v}}_I\right)}$$
We assume that the predicted true value appears at $j^{\ast }$ positions, the predicted output should be${\mathbf{t}}_{j^{\ast }}$Is a one-hot vector, and only $j^{\ast }$ positions have a$1$ , other positions are$0$ , then our purpose is to want to outputy \mathbf{y}jth of$y$$j^{The\; larger\; the\; value\; of\; the\; \ast }$ position, the better (as close as possible to$t_{j^{\ast }}$), the smaller the value of other positions, the better,
$$\begin{array}{rl}\max p\left(w_O\mid w_I\right)& =\max y_{j^{\ast }}\\ & =\max \log y_{j^{\ast }}\\ & =u_{j^{\ast }}-\log \sum _{j^{\prime }=1}^V\exp \left(u_{j^{\prime }}\right):=-E,\end{array}$$
For example, $E=-\log p\left(w_O\mid w_I\right)$ is the loss function, to maximize the above probability, that is to minimize$E.$ _
Therefore, the parameters can be updated by backpropagating the above formula as follows,
$$\begin{gathered} \frac{\partial E}{\partial w_{ij}^{\prime }}=\frac{\partial E}{\partial u_j}\cdot \frac{\partial u_j}{\partial w_{ij}^{\prime }}=(y_j-t_j)\cdot h_i \\ {{\mathbf{v}}_j^{\prime }}^{\text{(new)}}={v_j^{\prime }}^{\text{ (old)}}-the\cdot (y_j-t_j)\cdot h\text{ for }j=1,2,\cdots ,V \end{gathered}$$

$y_j,t_j,h_i$Represent vectors $\mathbf{y},\mathbf{t},h$ 's$j,j,i$ elements,$w_{ij}$Represents matrix ${\mathbf{W}}^{\prime }$ thiirow$i$$j$ column elements,$\eta$ is the learning rate.
Similarly, update$The\; process\; of\; W$ is as follows, we first seek$E$ to$h_i$ 的导数如下：
$$\frac{\partial E}{\partial h_i}=\sum _{j=1}^V\frac{\partial E}{\partial u_j}\cdot \frac{\partial u_j}{\partial h_i}=\sum _{j=1}^V(y_j-t_j)\cdot w_{ij}^{\prime }:={\mathrm{E}\mathrm{H}}_i$$

$:=$ means record as.

The parameter definitions are the same as above. The next step is to calculate $E$ vsWWThe partial derivative of$W$$W$ 'skkthAfter transposing$k$$h$ , is also said,$h_i=w_k$, where $k$ represents the$k$ elements are$1$ , others are$0$ , so we update$W$ only needs to update the$K$ rows, other values ​​are$0$，保持不变：
$$\frac{\partial E}{\partial w_k}=\frac{\partial E}{\partial h_i}\cdot \frac{\partial h_i}{\partial w_k}=\sum _{j=1}^V(y_j-t_j)\cdot w_{ij}^{\prime }={\mathrm{E}\mathrm{H}}_i\cdot x_k\text{for }w\text{ is a constant, }i=1,2,\cdots ,The\; N-$$
vector representation is updated as follows:
$${\mathbf{v}}_I^{(\text{new })}=v_I^{\text{(old) }}-\eta {\mathrm{E}\mathrm{H}}^T$$
also updates only$W$ 's$k$ line, other lines remain unchanged.

3.2 Multi-word context

Now we extend the model to the case where there are multiple context word inputs. The figure below shows a Multi-word context CBOW model. At this time, when

calculating the output of the hidden layer, the CBOW model does not directly copy the input vector of the input context, but is to take the average of the input context vectors and use as output the average vector of the product of the input to the hidden weight matrix:
$$\begin{array}{rl}\mathbf{h}& =\frac{1}{C}{\mathbf{W}}^T\left({\mathbf{x}}_1+{\mathbf{x}}_2+\cdots +x_C\right)\\ & =\frac{1}{C}{\left({\mathbf{v}}_{w_1}+{\mathbf{v}}_{w_2}+\cdots +v_{w_C}\right)}^T\end{array}$$
Among them, $C$ is the number of context words (in One-word context,$C=1$），$w_1,\cdots ,w_C$is the word vector of the context, ${\mathbf{v}}_1,\cdots ,v_C$ 同上。损失函数定义如下：
$$\begin{array}{rl}E& =-\log p\left(w_O\mid w_{I,1},\cdots ,w_{I,C}\right)\\ & =-u_{j^{\ast }}+\log \sum _{j^{\prime }=1}^V\exp \left(u_{j^{\prime }}\right)\\ & =-{\mathbf{v}}_O^{\prime }\cdot \mathbf{h}+\log \sum _{j^{\prime }=1}^V\exp \left({\mathbf{v}}_j^{\prime }{}^T\cdot \mathbf{h}\right)\end{array}$$
Since the part from the hidden layer to the output vector has not changed, so ${\mathbf{W}}^{\prime }$ Differentiation of the boundary value:
$${{\mathbf{v}}_j^{\prime }}^{\text{(new)}}={v_j^{\prime }}^{\text{ (old)}}-the\cdot (y_j-t_j)\cdot h\text{ for }j=1,2,\cdots ,VW$$
$The\; update\; of\; W$ is actually similar, except that we need to update each word$w_{I,c}$Definition:
$${\mathbf{v}}_{I,c}^{(\mathrm{n}\mathrm{e}\mathrm{w})}=v_{I,c}^{(\text{old })}-\frac{1}{C}\cdot the\cdot {\mathrm{E}\mathrm{H}}^T\text{ for }c=1,2,\cdots ,C$$
任利，$v_{I,c}$is the ccth in the input context$The\; input\; vector\; of\; c$ words, or the corresponding matrix$W$ 's$k_c$line, where $k_c$for input $c$ words are not 0 0in one-hot encoding$0$ 's dimension.

For example, in the context word, the first word $c=1$ is "I",$V=4$ , encoded as$\left[1000\right]$ , then,$k_1=0$ ; in the context word, the second word$c=2$ is "country",$V=4$ , encoded as$\left[0001\right]$ , then,$k_2=3$。

4. Skip-Gram Model

The figure below shows the structure of the Skip-Gram Model:

The derivation of the parameter update equation of the Skip-Gram Model is not much different from that of the one-word-context model. The loss function becomes:
$$\begin{array}{rl}E& =-\log p\left(w_{O,1},w_{O,2},\cdots ,w_{O,C}\mid w_I\right)\\ & =-\log \prod _{c=1}^C\frac{\exp \left(u_{c,j_c^{\ast }}\right)}{{\sum }_{j^{\prime }=1}^V\exp \left(u_{j^{\prime }}\right)}\\ & =-\sum _{c=1}^C{u}_{j_c^{\ast }}+C\cdot \log \sum _{j^{\prime }=1}^V\exp \left(u_{j^{\prime }}\right)\end{array}$$
Since the Skip-Gram Model is given an input word and predicts the probability of multiple words in its context, the loss function becomes $p\left(w_{O,1},w_{O,2},\cdots ,w_{O,C}|w_I\right)$ negative logarithm. where$j_c^{\ast }$is cc in the glossary$c$ context output words are not 0in one-hot encodingThe dimension of$0\; (explained\; as\; above).$

Since there are multiple outputs, we find the loss EEThe partial derivative of$E$
$$\frac{\partial E}{\partial u_{c,}}=y_{c,}-t_{c,}:=e_{c,}$$

$:=$ is to$y_{c,}-t_{c,}$denoted as $e_{c,}$

For simplicity, we write again:
$${EI}_j=\sum _{c=1}^C{e}_{c,}$$
As before, $c$ represents the number of output contexts. The next step is to seek$Let$ W ′ {\mathbf{W}}^\prime${\mathbf{W}}^{\prime }$ 中元素的偏导数，并更新：
$$\frac{\partial E}{\partial w_{ij}^{\prime }}=\sum _{c=1}^C\frac{\partial E}{\partial u_{c,}}\cdot \frac{\partial u_{c,}}{\partial w_{ij}^{\prime }}={EI}_j\cdot h_i$$
则更新如下：
$$w_{ij}^{\prime }(\text{ new })=w_{ij}^{\prime }\text{ (old) }-the\cdot {EI}_j\cdot h_i$$
Definition:
$${\mathbf{v}}_j^{\prime }\text{ (new) }=v_j^{\prime }\text{ (old) }-the\cdot {EI}_j\cdot h\text{ for }j=1,2,\cdots ,V$$
can see that, except for the loss function, there is almost no difference between the other and the one-word-context model.

Finally, find the update equation of the weight matrix from the input layer to the hidden layer. Since it can be seen from the structure that it is exactly the same as the one-word-context model, we directly give the update equation as follows:
$$v_I^{(\text{new })}={\mathbf{v}}_I^{(\text{old })}-the\cdot {\mathrm{E}\mathrm{H}}^T$$
where$EH$ is a$n-$ dimensional vector, each element in the vector is:
$${EH}_i=\sum _{j=1}^V{EI}_j\cdot w_{ij}^{\prime }$$

5. Finally

Definition:
$${{\mathbf{v}}_j^{\prime }}^{\text{(new)}}={v_j^{\prime }}^{\text{ (old)}}-the\cdot (y_j-t_j)\cdot \mathbf{h}\text{ for }j=1,2,\cdots ,V$$
word2vec needs all the words in the vocabulary on the variable side during the update process. For each step of the update, we also need to calculate$w_j,u_j,y_j,e_j$Finally, a $v_j^{\prime }$, the calculation amount can be imagined. All in the actual word2vec, a lot of tricks are used to optimize the calculation efficiency, and the details will not be expanded. I recommend the blog in References and this blog .

References

1. https://easyai.tech/ai-definition/word-embedding/#wordembedding
2. word2vec Parameter Learning Explained
3. https://mp.weixin.qq.com/s/7dsjfcOfm9uPheJrmB0Ghw