CS 224N Summary

CS 224N网址：Stanford CS 224N | Natural Language Processing with Deep Learning

Lecture1

PPT URL: PowerPoint Presentation (stanford.edu)

This lecture mainly talks about the object of NLP research, how we represent the meaning of words, and the basic principles of the Word2Vec method.

Here we briefly introduce some basic principles of the Word2Vec method: people think that a word often has the same meaning as its context words, so the expression vectors of these words are expected to be similar. We define the words in a window as the center word and the context word, as follows As shown in the figure.

The loss function can be written as:

• $J(\theta )=-\frac{1}{T}\log L\left(i\right)=-\frac{1}{T}{\sum }_{t=1}^T{\sum }_{\begin{array}{c}m\le j\le m\\ j\ne 0\end{array}}\log P\left(w_{t+j}\mid w_t;i\right)$

  • Loss is rewritten as a logarithm for the convenience of calculation, so that all multiplications will become additions

• Why use two vectors per word? (It is mentioned in the PPT of Lecture2, but not in detail)

  • For the convenience of mathematical calculations
    • softmax: $P(o|c)=\frac{\exp \left(u_o^T{v}_c\right)}{{\sum }_{w\in V}\exp \left(u_w^T{v}_c\right)}$
    • It can be noticed that in the denominator, there is such a term ${\sum }_{w\in V}{u}_w^T{v}_c$, if we compare it to $v_c$Derivation, the result is ${\sum }_{w\in V}{u}_w$. However, if we do not use two sets of vectors, the term should be written as ${\sum }_{w\in V}{v}_w^T{v}_c$, what needs to be noted here is that $w$ may be the same as$c$ is the same, we can write${\sum }_{w\in V,w\ne c}{v}_w^T{v}_c+v_c^T{v}_c$, for $v_c$Derivation, the result is ${\sum }_{w\in V,w\ne c}{v}_w+2v{\_}_c$, which leads to a result that is not as easy to understand as the result obtained using the two vectors.
  • In the end, the two vectors will be very similar, but not the same. We take the average of the two vectors to express the final word vector
  • Specific derivation: 01 Introduction and Word Vectors - The Sun Also Rises

Lecture2

PPT URL: cs224n-2023-lecture02-wordvecs2.pdf (stanford.edu)

Bag of words model: The model does not consider the order of words, and the prediction of a word in different positions is the same

• Two variants of Word2Vec:
  • Skip-grams: Given the center word to predict the context word (the Word2Vec model we showed is this form)
  • Continuous Bag of Words (CBOW): Predict the central word through the context word

Methods that can be used to update:

• Gradient Descent (GD): After calculating the gradient on all samples, update it again
• Stochastic Gradient Descent (SGD): update with a single sample at a time
• Mini-batch Gradient Descent (MBGD): Each time a batch of samples are used to update, between the above two

Available for training loss function:

• Simple softmax(simple, but when there are many categories, the calculation is very heavy)
• Optimized variants such as hierarchicalsoftmax
• Negative sampling ( negative sampling)

Above, we are using loss functionthe naivesoftmax

• $J(\theta )=-\frac{1}{T}\log L\left(i\right)=-\frac{1}{T}{\sum }_{t=1}^T{\sum }_{\begin{array}{c}m\le j\le m\\ j\ne 0\end{array}}\log P\left(w_{t+j}\mid w_t;i\right)$

  • $P(w_{t+j}|w_t;i)=P(o|c)$ , the denominator of this term is very computationally intensive, so in the standardword2vec, we do not use this form, but use the negative sampling method

  • The core idea of ​​negative sampling: train a binary logistic regression to distinguish a true pair (center word and words in the context window) from some noise pairs (center word and some random words)

Negative sampling loss function:

• 最大化 $J_t\left(i\right)=\log p\left(u_o^T{v}_c\right)+{\sum }_{i=1}^k{\mathbb{E}}_{j\sim P(w)}\left[\log p\left(-u_j^T{v}_c\right)\right]$
  • We maximize the probability of two words co-occurring in the first log and minimize the probability of noise words in the second log
  • k represents the number of negative samples sampled
• 这可以写成 J neg-sample  ( u o , v c , U ) = − log ⁡ σ ( u o T v c ) − ∑ k ∈ { K  sampled indices  } log ⁡ σ ( u k T v c ) J_{\text {neg-sample }}\left(\boldsymbol{u}_o, \boldsymbol{v}_c, U\right)=-\log \sigma\left(\boldsymbol{u}_o^T \boldsymbol{v}_c\right)-\sum_{k \in\{K \text { sampled indices }\}} \log \sigma\left(\boldsymbol{u}_k^T \boldsymbol{v}_c\right) Jneg-sample ​(uo​,vc​,U)=−logp(uoT​vc​)−∑k∈{ K sampled indices }​logp(ukT​vc​)
  • Using the probability distribution $P(w)=U(w{)}^{3/4}/Z$to sample,$U\left(w\right)$ is a unigram distribution

---

The above mainly introduces the Word2Vec model, which is a prediction model (Direct prediction) based on the local context window. For learning word vector, there is another type of model that is count based global matrix factorization

• The advantage of Direct Prediction is that it can summarize more complex information than correlation, and the disadvantage is that it does not fully utilize statistical information
• The advantage of the Count based model is that it is trained quickly and effectively uses statistical information. The disadvantage is that it is biased towards high-frequency words and can only summarize the relevance of phrases.

count based model: construct a vocabulary co-occurrence matrix, each row is a word, and each column is context. But the matrix is ​​easy to be too large. We hope that the dimension of the word vector will not be too large, so we use some dimensionality reduction methods to learn the low-dimensional representation of word.

• Typical is SVD

• GloVe (Global Vectors): A method that combines the local context window and (count based) vocabulary co-occurrence matrix

  • For a detailed introduction, see: CS224N Notes (2): GloVe - Zhihu (zhihu.com)