Word Vector: GloVe Model Explained

  This content mainly introduces the GloVe model for constructing word vectors .

1 Introduction

  Before the GloVe model was proposed, there were two main types of models for learning word vectors:

• Global matrix factorization methods such as Latent semantic analysis (LSA).
• Local context window methods, such as the skip-gram model proposed by Mikolov et al.

  However, both methods have obvious drawbacks. While methods like LSA exploit statistics effectively, they perform relatively poorly on the word analogy task, suggesting a suboptimal vector space structure. Methods like skip-gram may do better on the analogy task, but they make little use of corpus statistics because they train on individual local context windows rather than global co-occurrence counts.

2 GloVe model

  The full name of GloVe is Global Vectors for Word Representation, which is a word representation tool based on count-based & overall statistics.

  The implementation of GloVe is mainly divided into three steps: (1) constructing the co-occurrence matrix; (2) the approximate relationship between the word vector and the co-occurrence matrix; (3) constructing the loss function.

2.1 Construction of co-occurrence matrix

  Suppose we have a corpus containing the following three sentences:

i like deep learning

i like NLP

i enjoy flying

  This corpus involves 7 words: i, like, enjoy, deep, learning, NLP, flying.

  Assuming that we use a statistical window with a size of 3 (left and right lengths are 1), taking the first statement " i like deep learning " as an example, the following window content will be generated:

window label	center word	window content
0	i	i love
1	love	i love deep
2	deep	love deep learning
3	learning	deep learning

  Taking window 1 as an example, the central word is love , and the context words are i and deep , then update the elements in the co-occurrence matrix:

$$\begin{gathered} X_{love,i}+=1 \\ {X}_{love,deep}+=1 \end{gathered}$$

  Using the above method, traverse the entire corpus to get the co-occurrence matrix $X$：

	i	like	enjoy	deep	learning	NLP	flying
i	0	2	1	0	0	0	0
like	2	0	0	1	0	1	0
enjoy	1	0	0	0	0	0	1
deep	0	1	0	0	1	0	0
learning	0	0	0	1	0	0	0
NLP	0	1	0	0	0	0	0
flying	0	0	1	0	0	0	0

Among them, the first column represents the central word, and the first row represents the context word.

2.2 Approximate relationship between word vector and co-occurrence matrix

  Before we start, let's define some variables:

• $X_{ij}$Denotes the word $j$ in wordiiThe number of occurrences of$i\; in\; the\; context.$
• $X_i={\sum }_k{X}_i$Indicates any word that occurs in the word iiThe number of times in the$i\; context.$
• $P_{ij}=P(j|i)=X_{ij}/X_i$Denotes the word $j$ occurs in the wordiiProbability in context of$i\; .$

  Let's take a look at a table provided by the author of the paper:

Table 1. Co-occurrence probabilities of the target words ice and steam with selected context words from the 6 billion corpus. It is only in this ratio that the noise from non-distinguished words is canceled out, so large values ​​(much larger than 1) correlate with properties of ice, and small values ​​(much smaller than 1) correlate with properties of steam.

  Table 1 shows the probabilities and their ratio results for a large corpus, where $i=ice$ and$j=steam$ . Forkkicebutnotsteam$k$ , such as$k=solid$，$P_i/P_{jk}$much greater than 1. Similarly, for a word kk related to steam but not ice$k$ , such as$k=gas$ , the ratio is much smaller than 1. Forwordswaterorfashion$k$ , which is either related toiceandsteamor neither, has a ratio close to 1. Compared to the original probabilities, this ratio is able to better distinguish related words (solidandgas) from irrelevant words (waterandfashion), and better distinguish two related words.

  The above argument shows that it is a more appropriate method to learn word embeddings through the ratio of probabilities rather than probabilities themselves. Ratio $P_i/P_{jk}$depends on three words $i$，$j$ and$k$ , the most general model takes the form:

$$F(w_i,w_j,{\tilde{w}}_k)=\frac{P_i}{P_{jk}}\text{\hspace{1em}(1)}$$

where $w\in {\mathbb{R}}^d$ is a word vector,$\tilde{w}\in R^d$ is the separate context word vector. In this equation, the right-hand side is extracted from the corpus,$F$ may depend on some yet-to-be-determined parameters. Because the vector space has a linear structure, the most natural way to express the proportional difference between two probabilities is to use the vector difference, which can be obtained:

$$F(w_i-w_j,{\tilde{w}}_k)=\frac{P_i}{P_{jk}}\text{\hspace{1em}(2)}$$

  Next, we notice that $The\; parameter\; of\; F$ is a vector, and the right side is a scalar, so we put$The\; parameters\; of\; F$ are subjected to dot product operation to obtain:

$$F((w_i-w_j{)}^T{\tilde{w}}_k)=\frac{P_i}{P_{jk}}\text{\hspace{1em}(3)}$$

  we know $X$ is a symmetric matrix, words and context words are actually relative, that is, if we do the following exchange:$w\leftrightarrow \tilde{w}$，$X\leftrightarrow X^T$ , formula (3) should remain unchanged. Obviously, the current formula is not satisfied. However, symmetry can be achieved in two steps. First, we ask$F$ satisfies the homomorphic property, namely:

$$F\left(\right(w_i-w_j{)}^T{\tilde{w}}_k)=\frac{F(w_i^T{\tilde{w}}_k)}{F(w_j^T{\tilde{w}}_k)}\text{\hspace{1em}(4)}$$

Combined with equation (3) we get:

$$F\left(w_i^T{\tilde{w}}_k\right)=P_i=\frac{X_i}{X_i}\text{\hspace{1em}(5)}$$

Let $F=\exp$ , get:

$$w_i^T{\tilde{w}}_k=\log (P_i)=\log (X_i)-\log \left(X_i\right)\text{\hspace{1em}(6)}$$

Next, we notice that if not log ⁡ ( X i ) \log(X_i)   on the right$\log \left(X_i\right)$ , Equation (6) will exhibit exchange symmetry. However, this term is not related to$k$ is irrelevant, so it can be absorbed as$w_i$The bias term $b_i$. Finally, for ${\tilde{w}}_k$Add a bias ${\tilde{b}}_k$Achieve symmetry:

$$w_i^T{\tilde{w}}_k+b_i+{\tilde{b}}_k=\log (X_i)\text{\hspace{1em}(7)}$$

  In this way, we get an approximate relationship between word vectors and co-occurrence matrices.

2.3 Construct loss function

  In the paper, the author proposes a new weighted least squares regression model to construct the loss function. Treat equation (7) as a least squares problem and introduce a weight function f ( X ij ) f(X_{ij}) in the loss function$f(X_{ij})$：

$$J=\sum _{i,j=1}^{V}f(X_{ij})(w_i^T{\tilde{w}}_j+b_i+{\tilde{b}}_j-\log X_{ij}{)}^2\text{\hspace{1em}(8)}$$

where $V$ is the size of the vocabulary.

  Because rare co-occurrences carry noise and carry less information than frequent co-occurrences. Adding a weight function avoids giving equal weight to all co-occurrence events. Thus the weight function should conform to the following properties:

• $f(0)=0$ . if$f$ is considered as a continuous function, it should increase as$x\to 0$ disappears fast enough that${\lim }_{x\to 0}f(x){\log }^{2}x$ is finite.

• $f\left(x\right)$ should be non-decreasing so that rare co-occurrences are not overweighted.

• for larger $x$ value,$f\left(x\right)$ should be relatively small so that frequent co-occurrences are not overweighted.

  Many functions satisfy these properties, and the authors found that functions of the following form perform well:

$$f\left(x\right)=\{\begin{array}{c}(x/x_{max}{)}^a,if(x<x_{max})\\ 1,otherwise\end{array}\text{\hspace{1em}(9)}$$

Figure 1 Weighting function f of α=3/4

The author uses xmax x_{max}   in all experiments$x_{max}$Fixed to 100. And found $a=3/4$ ratio$a=1$ with some improvements over the linear version.

3 Summary

  GloVe is a new global log-bilinear regression model for unsupervised learning of word vectors. It combines the advantages of both global matrix factorization and local context window methods. GloVe models make efficient use of statistics by training only on the non-zero elements in the word-word co-occurrence matrix, rather than the entire sparse matrix or a single context window in a large corpus. The GloVe model generates a vector space with meaningful substructure.

3.1 The difference between GloVe and LSA

  LSA is an earlier statistics-based word vector representation tool. It is also based on co-occurrence matrix, but it uses matrix decomposition technology based on Singular Value Decomposition (SVD) to reduce the dimensionality of large matrices, because the complexity of SVD is very large. High, so its computational cost is relatively large. Another point is that it has the same statistical weight for words. These shortcomings are overcome in GloVe.

3.2 The difference between GloVe and word2vec

  Glove and word2vec mainly have the following differences:

• word2vec uses the neural network to predict the central word (CBOW) according to the context or the context (skip-gram) according to the central word; GloVe is a logarithmic bilinear regression model, which essentially reduces the dimensionality of the co-occurrence matrix.
• Both use sliding windows, but word2vec is used directly for training, while GloVe is used to build co-occurrence matrices.
• word2vec only pays attention to local information, GloVe utilizes local information and global information.
• GloVe is faster to train than word2vec.
  
  

reference

[1] GloVe: Global Vectors for Word Representation

[2] Stanford University's word vector tool: GloVe

[3] (15) Easy to understand - the principle of Glove algorithm

[4] GloVe Homepage