Natural language processing - word2vec project combat - from Word2Vec to FastText

From Word2Vec to FastText

Application of Word2Vec in Deep Learning

• Text generation (Word2Vec + RNN/LSTM)
• Text Classification (Word2Vec + CNN)

text generation

Neural network: a nonlinear regression model composed of a bunch of formulas

common neural network

neural network with memory

Therefore, direct feed alone is not enough. We hope that our classifier can remember the contextual relationship:

The purpose of RNN is to allow information with sequential relationships to be considered.

What is a sequential relationship? It is the context of information in time.

RNN

SS in each time point$S$ calculation: short-term memory
$$S_t=f(U_{x_t}+W_{s_{t-1}})$$
The final output of this neuron, based on the last$S$
$$O_t=softmax(V_{S_t})$$
Simply put, for t=5, it is actually equivalent to stretching one neuron into five

In other words, S is what we call memory (because the information of t from 1-5 is recorded)

LSTM

long short-term memory —— long-term short-term memory

The most important thing in LSTM is the Cell State, which goes all the way down and runs through this timeline, representing the bond of memory.

It will be messed with by XOR and AND operators to update the memory

The control of the increase and decrease of information depends on these valves: Gate

1 means, remember all the information of this trip

0 means, the information of this trip can be forgotten

1. forget the door

to decide what information we should forget

It compares the last state ht-1 with the current input xt. Output a value from 0 to 1 through the gate (just like an activation function)

1 represents: Remember it for me! 0 means: forget it!

$$f_t=s(w_f\cdot [h_{t-1},x_t]+b_f)$$

2. memory door

what to remember

This door is more complicated, divided into two steps:

The first step, use sigmoid to decide what information needs to be updated by us (forget the old)

The second step is to create a new Cell State (updated cell state) with Tanh

$$\begin{gathered} i_t=s(W_i\cdot [h_t-1,x_t]+b_i) \\ \widetilde{C_t}=tanh(W_C\cdot [h_{t-1},x_t]+b_C) \end{gathered}$$

3. update gate

Update the old cell state to the new cell state

Update our cell state with gates like XOR and AND:

$$C_t=f_t\ast C_{t-1}+i_t\ast \widetilde{C_t}$$

4. output gate

memory to determine what value to output

Our Cell State has been updated,

So we use this memory link to determine our output:

(The Ot here is similar to the output we just ran out of the RNN in one step)

$$\begin{gathered} O_t=s\left(W_{}\right[h_{t-1},x_t]+b_o) \\ {h}_t=O_t\ast tanh\left(C_t\right) \end{gathered}$$

Text Categorization

Baseline: BoW + SVM

Deep Learning: CNN for Text

CNN4Text

Blur + Sharpen

How to migrate to word processing?

$$C_i=f(W^T{X}_{i:i+h-1}+b)$$

1. convert text into pictures
   

2. Make CNN into 1D

RNN generates text——→ strict logic, order guaranteed

CNN Text Classification —— → error tolerance

Boundary handling:

Narrow vs Wide

Stride size: (sweep in a few steps)

Fast Text

word2vec:

Fast Text:

1. Bow ——→ Bi-Gram
2. Hashing Trick
3. Hierachy Softmax

Focus on text classification, space compression and time acceleration