Natural language processing from entry to application - dynamic word vector pre-training: bidirectional language model

Category: General Catalog of "Natural Language Processing from Entry to Application"

---

For a given piece of input text $w_1{w}_2\cdots w_n$, the bidirectional language model builds the language model simultaneously from the forward (from left to right) and backward (from right to left) directions. The advantage of this is that for any word $w_t$, the representations based on the left context information and the right context information respectively can be obtained at the same time. Specifically, the model first encodes each word individually. This process is context-independent and mainly uses the character sequence information inside the word. Based on the encoded word representation sequence, the model uses two multi-layer long short-term memory networks (LSTM) in different directions to calculate the forward and backward hidden layer representations of the words at each moment, that is, the context-dependent word vector representation. Using this representation, the model predicts the target word at each moment. For the forward language model, ttThe target word at time$t$$w_{t+1}$, for the backward language model, the target word is $w_{t-1}$。

input presentation layer

The ELMo model uses a character combination-based neural network to represent each word in the input text, with the purpose of reducing the impact of Out-Of-Vocabulary (OOV) on the model. The figure below shows the basic structure of the input representation layer. First, the character vector layer converts each character in the input layer (with additional start and stop characters) into a vector representation. Suppose $w_t$From the character sequence $c_1{c}_2\cdots c_l$constitute, for each character ci c_i in it$c_i$, can be expressed as: $v_{c_i}=E^{\text{char}}{e}_{c_i}$。其中，$E^{\text{char}}\in R^{d^{\text{char}}\times |V^{\text{char}}|}$represents a character vector matrix;$V^{\text{char}}$ represents all character sets;$d^{\text{char}}$ represents the character vector dimension;$e_{c_i}$Indicates the character $c_i$one-hot encoding of . remember $w_t$The matrix composed of all character vectors in is $C_t\in R^{d^{\text{char}}\times l}$，即$C_t=[v_{c_1},v_{c_2},\cdots ,v_{c_l}]$ . Next, use the convolutional neural network to perform semantic composition (Semantic Composition) on the character-level vector representation sequence. A one-dimensional convolutional neural network is used here, and the dimension of the character vector$d^{\text{char}}$ is used as the number of input channels, recorded as$N^{\text{in}}$ , the dimension of the output vector is used as the number of output channels, recorded as$N^{\text{out}}$ . In addition, by using multiple convolution kernels of different sizes (widths), character-level context information of different granularities can be utilized, and corresponding hidden layer vector representations can be obtained. The dimensions of these hidden layer vectors are corresponding to each convolution kernel The number of output channels is determined. By splicing these vectors, the convolution output of each position is obtained. Then, the output vectors of all positions of the hidden layer are pooled to obtain the word$w_t$The fixed-length vector representation of , denoted as $f_t$. Assuming that 7 one-dimensional convolution kernels with widths {1, 2, 3, 4, 5, 6, 7} are used, the corresponding output channels are {32, 32, 64, 128, 256, 512, 1024 }, then the output vector $f_t$has a dimension of 2048. Then, the model uses a two-layer Highway neural network to further transform the output of the convolutional neural network to obtain the final word vector representation $x_t$. The Highway neural network directly establishes a "channel" between the input and the output, so that the output layer can directly transmit the gradient back to the input layer, thereby avoiding the problem of gradient explosion or dispersion caused by too many network layers. The specific calculation method of the single-layer Highway neural network is as follows:
$$x_t=g\odot f_t+(1-g)\odot \text{ReLU}(W{f}_t+b)$$

where, $g$ is the gating vector, which is$f_t$As the input, it is calculated by the Sigmoid function after linear transformation:
$$g=s(W^g{f}_t+b^g)$$

In the formula, $W^g$ and$b^g$ is the linear transformation matrix and bias vector in the gating network. It can be seen that the output of the Highway neural network is actually the result of linear interpolation between the input layer and the hidden layer. Of course, the structure of the model is usually adjusted and determined according to the experiment, and we can also try other model structures by ourselves. For example, a sequence of intra-word strings can be encoded using a character-level bidirectional LSTM network. Next, on the basis of the context-free word vectors obtained by the above process, a bidirectional language model is used to encode the forward and backward context information respectively, so as to obtain the dynamic word vector representation at each moment.

forward language model

In the forward language model, the prediction of the target word at any moment only depends on the context information or history on the left side of the moment. Here we use a long short-term memory network language model based on multi-layer stacking. Record the parameters of the multi-layer stacked LSTM in the model as ${\overrightarrow{i}}^{\text{LSTM}}$ , Softmax output layer parameters are recorded as$i^{\text{out}}$ . Then the model can be expressed as:
$$p(w_1{w}_2\cdots w_n)=\prod _{t=1}^{n}P(w_t|x_{1:t-1};{\overrightarrow{i}}^{\text{LSTM}};i^{\text{out}})$$

backward language model

Contrary to the forward language model, the backward language model only considers the context information on the right at a certain moment. Can be expressed as:
$$p\left(w_1{w}_2\cdots w_n\right)=\prod _{t=1}^{n}P(w_t|x_{t+1:n};{\overleftarrow{i}}^{\text{LSTM}};i^{\text{out}})$$

References:
[1] Che Wanxiang, Cui Yiming, Guo Jiang. Natural language processing: a method based on pre-training model [M]. Electronic Industry Press, 2021. [2] Shao Hao, Liu Yifeng. Pre-
training language model [M] ]. Electronic Industry Press, 2021.
[3] He Han. Introduction to Natural Language Processing [M]. People's Posts and Telecommunications Press, 2019 [
4] Sudharsan Ravichandiran. BERT Basic Tutorial: Transformer Large Model Combat [M]. People's Posts and Telecommunications Publishing Society, 2023
[5] Wu Maogui, Wang Hongxing. Simple Embedding: Principle Analysis and Application Practice [M]. Machinery Industry Press, 2021.