[Paper Interpretation Series] NER Direction: LatticeLSTM (ACL2018)

Introduction

LatticeLSTM comes from Chinese NER Using Lattice LSTM in ACL2018.

Paper address:
https://arxiv.org/abs/1805.02023

There are multiple versions of the code :
Official version: https://github.com/jiesutd/LatticeLSTM
Reproduced by others: https://github.com/LeeSureman/Batch_Parallel_LatticeLSTM

The LSTM-CRF model has a significant effect in the English named entity recognition task. In the Chinese NER task, the character-based NER model is also significantly better than the word-based NER model (to avoid the impact of word segmentation errors on the NER task). Introducing lexical information into the character-based NER model to determine entity boundaries can significantly improve the Chinese NER task.

The Lattice LSTM model is the beginning of the Chinese NER based on the vocabulary enhancement method . In this model, character information and all word sequence information are used . Specifically, when we match a sentence through lexical information (dictionary), a Lattice-like structure can be obtained. This method can avoid entity recognition errors caused by word segmentation errors, and has a significant effect on Chinese NER tasks.

model structure

LSTM structure

LSTM is a variant of RNN, which can effectively solve the problem of gradient disappearance and gradient explosion. Three gates are mainly introduced, namely the input gate $i_t$, the forget gate $f_t$, the output gate $o_t$And with a new Cell State $c_t$Linear transmission of information, while non-linear output information to the Hidden State $h_t$。

Form:
$$\begin{array}{rl}& \left[\begin{array}{c}{i}_t\\ o_t\\ f_t\\ {\tilde{c}}_t\end{array}\right]=\left[\begin{array}{c}p\\ p\\ p\\ \end{array}\right]\left(W\left[\begin{array}{c}{X}_t\\ h_{t-1}\end{array}\right]+b\right)\\ & c_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t\\ & h_t=o_t\odot \mathrm{fishy}\left({\mathbf{c}}_t\right)\end{array}$$

It can be seen from the above formula:

• Input gate ${\mathbf{i}}_t$Used to control the current candidate state ${\tilde{\mathbf{c}}}_t$How much information needs to be saved.

• Forget gate ${\mathbf{f}}_t$Used to control the previous state $c_{t-1}$How much information needs to be forgotten.

• output gate ${\mathbf{o}}_t$The user controls the state of the current moment $c_t$How much information needs to be output to ${\mathbf{h}}_t$

The article introduces three types of model schemes, including Character-Based Model, Word-Based Model and Lattice Model, but its main network structure is LSTM-CRF.

Character-Based Model

For the Character-Based model, the input is the character sequence $c_1,c_2,\dots ,c_m$, which are directly input to LSTM-CRF. where each character $c_j$表征的${\mathbf{x}}_j^c={\mathbf{e}}^c\left(c_j\right)，{\mathbf{e}}^c$ is the embedding matrix of the character, that is, the lookup table of the character representation. Usually bidirectional LSTM is used to characterize the input sequence$x_1,x_2,\dots ,x_m$Process to get the Hidden State sequence from left to right and right to left ${\overrightarrow{\mathbf{h}}}_1^c,{\overrightarrow{h}}_2^c,\dots ,{\overrightarrow{h}}_m^c$ 和 ${\overleftarrow{\mathbf{h}}}_1^c,{\overleftarrow{h}}_2^c,\dots ,{\overleftarrow{h}}_m^c$. Then splicing the character representations in the two directions, that is, the Hidden State of each character is expressed as: $h_j^c=\left[{\overrightarrow{h}}_j^c;{\overleftarrow{h}}_j^c\right]$
最后在$h_1^c,h_2^c,\dots ,h_m^c$Then a standard CRF layer is followed for sequence annotation. superscript $c$ means character level.

In addition, the Character-Based model can also integrate n-gram information (char+bichar) and word segmentation information (char+softword).

• Char + bichar: directly splicing the embedded representation of bigram and the embedded representation of char as the embedded representation of the character.
• Char + softword: Word segmentation information can be added to the Character-Based model as a soft feature. Concatenate word segmentation label embedding representation and character embedding representation xjc = [ ec ( cj ) ; es ( seg ⁡ ( cj ) ) ] \mathbf{x}_{j}^{c}=\left[\mathbf{e} ^
  $${\mathbf{x}}_j^c=\left[{\mathbf{e}}^c\left(c_j\right);{\mathbf{e}}^s\left(\mathrm{say}\left(c_j\right)\right)\right]$$
  es$e^s$ is the embedding matrix represented by the word segmentation tag embedding,$seeg\left(c_j\right)$ means that for$c_j$The word segmentation tag results of characters mainly include 4 kinds of tag results of BMES, indicating the position of the word segmentation result of the character.

Word-Based Model

The Word-Based model is similar to the Character-Based model. The input is a word sequence $w_1,w_2,...,w_n$，Every sentence $w_i$Expression: ${\mathbf{x}}_i^w={\mathbf{e}}^w(w_i)$ , among which$e^w$ is the word embedding matrix. In the same way, it also inputs a bidirectional LSTM, and then concatenates the two results as the output Hidden State. Sequence annotation is achieved by concatenating softmax and CRF.

Word-Based can further fuse character information. word $w_i$The character representation in $x_i^c$, take the splicing result of the two as the new representation of the word:
$$x_i^w=[{\mathbf{e}}^w\left(w_i\right);x_i^c]$$
As for how to get the word$w_i$The characters in ${\mathbf{x}}_i^c$, there are three ways:

• Word + char LSTM: Directly input each character into the bidirectional LSTM, and splicing the hidden states in both directions as the word $w_i$的字符表征：
  $$x_i^c=\left[{\overrightarrow{\mathbf{h}}}_{t(i,\mathrm{len}(i))}^c;{\overleftarrow{h}}_{t(i,1)}^c\right]$$
  of which$t(i,k)$ means the$The\; index\; of\; i$ words is$The\; character\; of\; k$ , so$t(i,len(i))$ display lyrics$w_i$the last character of the . It can be seen that this method actually only takes the splicing results of the first and last characters.

• Word + char LSTM': Different from the above method, this method uses an LSTM to obtain each character $c_j$The two-way representation of the character is spliced ​​together as the hidden states representation of the character, and finally integrated into the word representation. This method is similar to the structure of Liu et al. (2018) but does not use the highway layer.

• Word + char CNN: Use CNN to extract the character sequence representation of each word, character representation ${\mathbf{x}}_i^c$, character $c_j$The word vector uses ${\mathbf{e}}^c(c_j)$ means:

where ${\mathbf{W}}_{\mathbf{CNN}}$和${\mathbf{b}}_{\mathbf{CNN}}$are parameters to be learned, $ke$ means kernal size, the value in the text is 3,$maxmeans$ max pooling.

Lattice Model

Lattice LSTM can be seen as an extension of the Character-Based model, which adds words as input (for example, the input in the figure below includes "Nanjing City", use the red part of the network structure to extract features) and additional gates (green in the figure below Gating units connected by wires) to control the flow of information.

The input of Lattice LSTM is: character sequence $c_1,c_2,...,c_m$, with the dictionary DDAll character subsequences matched by words in$D$$w_{b,e}^d$，$b$ is the subscript of the starting character,$e$ is the subscript of the ending character. For example, for the word segmentation result of the sentence "Nanjing Yangtze River Bridge" in Figure 1,$w_{1,2}^d$Means "Nanjing". The input of the model has 4 kinds of vectors: input vector, hidden layer output vector, cell vector, and gating vector. The input vector is a character vector , ${\mathbf{x}}_j^c={\mathbf{e}}^c(c_j)$ . The recursive structure of the LSTM model is reflected in each character$c_j$Use a character cell vector $c_j^c$And a hidden layer vector ${\mathbf{h}}_j^c$, where $c_j^c$Used to record from the beginning of the sentence to the character $c_j$The information flow will eventually $h_j^c$Input to the CRF layer for sequence annotation.

Unlike the character-based model, calculate $c_j^c$When considering the substring $w_{b,e}^d$, each substring $w_{b,e}^d$的表征的：${\mathbf{x}}_{b,e}^w={\mathbf{e}}^w(w_{b,e}^d)$ . In addition,the word cell State ${\mathbf{c}}_{b,e}^w$Represents the recursive state xb starting from the sentence , ew \mathbf{x}^w_{b,e}$x_{b,e}^w$, calculated as follows:

$$\begin{array}{rl}\left[\begin{array}{c}{\mathbf{i}}_{b,e}^w\\ {\mathbf{f}}_{b,e}^w\\ {\tilde{c}}_{b,e}^w\end{array}\right]& =\left[\begin{array}{c}p\\ p\\ \end{array}\right]\left({\mathbf{W}}^{w\top }\left[\begin{array}{c}{\mathbf{x}}_{b,e}^w\\ {\mathbf{h}}_b^c\end{array}\right]+b^w\right)\\ c_{b,e}^w& ={\mathbf{f}}_{b,e}^w\odot {\mathbf{c}}_b^c+{\mathbf{i}}_{b,e}^w\odot {\tilde{c}}_{b,e}^w\end{array}$$
Among them, $i_{b,e}^w$和$f_{b,e}^w$They are the input gate and the forget gate. ${\mathbf{h}}_b^c$和$c_b^c$They are the Hidden State and Cell State output from the first word LSTM unit of the word, such as calculating 长江大桥the cell state of the word, $h_b^c$和$c_b^c$From 长the word, and the calculated 大桥cell State, $h_b^c$和$c_b^c$from 大the word. In addition, it should be noted that there is no output gate for word-level cells, because the labeling results of sequence labeling are all at the character level.

Cell status cb by word , ew \mathbf{c}^w_{b,e}$c_{b,e}^w$There will be more recursive path information flowing into each $c_j^c$. For example, in Figure 2, the input source $c_7^c$including ${\mathbf{x}}_7^c$(桥)、${\mathbf{c}}_{6,7}^w$(Bridge), $c_{4,7}^w$(Yangtze River Bridge). Put all $c_{b,e}^w$link to cell $c_e^c$, that is, weighted addition of these three parts of information. For the cell status cb of each substring $c_{b,e}^w$Using an extra gate ${\mathbf{i}}_{b,e}^c$to control the flow into $c_e^c$The information flow (the original text is $c_{b,e}^c$, should be a typo):

Compute character $c_j^c$cell state $c_j^c$, that is, for all $c_j^c$The cell state and cjc c_j^c of the ending word$c_j^c$Candidate state ${\tilde{c}}_j^c$Do a weighted addition. Therefore the characters $c_j^c$的cell state $c_j^c$ 的计算如下：
$$c_j^c=\sum _{b\in \left\{b^{\prime }\mid w_{b^{\prime },j}^d\in \mathbb{D}\right\}}{a}_{b,j}^c\odot c_{b,j}^w+a_j^c\odot {\tilde{c}}_j^c$$

α \alpha in the above formula$\alpha$ is the result of gating normalization, and the gating is based on the cell state${\mathbf{c}}_{b,e}^w$and character embedding ${\mathbf{x}}_e^c$Solution:
$${\mathbf{i}}_{b,e}^c=p\left({\mathbf{W}}^{l\top }\left[\begin{array}{c}{x}_e^c\\ c_{b,e}^w\end{array}\right]+{\mathbf{b}}^l\right)$$

$i_{b,j}^c$ 和 $i_j^c$Respectively normalized to $a_{b,j}^c$and $a_j^c$, so that it sums to $1$：

$$\begin{array}{rl}{a}_{b,j}^c& =\frac{\exp \left(i_{b,j}^c\right)}{\exp \left({\mathbf{i}}_j^c\right)+{\sum }_{b^{\prime }\in \left\{b^{\prime \prime }\mid w_{b^{\prime \prime },j}^d\in \mathbb{D}\right\}}\exp \left(i_{b^{\prime },j}^c\right)}\\ a_j^c& =\frac{\exp \left(i_j^c\right)}{\exp \left(i_j^c\right)+{\sum }_{b^{\prime }\in \left\{b^{\prime \prime }|w_{b^{\prime \prime },j}^d\in D\right\}}\exp \left(i_{b^{\prime },j}^c\right)}\end{array}$$

Taking 南京市长江大桥as an example, the calculated 桥cell state is calculated by weighting and 长江大桥summing the 大桥cell state of , and the candidate cell state of . 桥The weighting coefficient is obtained by performing Softmax on the gating of characters and words.
It should be pointed out that when the current character has vocabulary integrated, the above formula is used for calculation; if the current character has no vocabulary, the native LSTM is used for calculation. In other words, when there is vocabulary information , Lattice LSTM does not use the memory vector cj − 1 c c_{j-1}^c of the previous moment$c_{j-1}^c$, that is, no continuous memory of lexical information is retained.

Decoding 和 Training

The Decoding part is in ${\mathbf{h}}_1,h_2,...,h_{}$Then follow a CRF layer to calculate the output probability, and then select the largest output probability argmaxP(y|s). For Character-Based Model, $\tau$ value is$n$ , for Word-Based Model,$\tau$ value is$m$ . The following is to calculate a label sequence$probability\; of\; y$ .

$$P(and\mid s)=\frac{\exp \left({\sum }_i\left(W_{CRF}^{l_i}{h}_i+b_{CRF}^{\left(l_{i-1},l_i\right)}\right)\right)}{{\sum }_{y^{\prime }}\exp \left({\sum }_i\left({\mathbf{W}}_{\mathrm{CRF}}^{l_i^{\prime }}{\mathbf{h}}_i+b_{CRF}^{\left(l_{i-1}^{\prime },l_i^{\prime }\right)}\right)\right)}$$

Use the first-order Viterbi algorithm to find the highest-scoring label sequence, and the loss function is log-likelihood loss with L2 regularization.

$$L=\sum _{i=1}^N\log \left(P\left(y_i\mid s_i\right)\right)+\frac{}{2}\Vert \Theta {\Vert }^2$$

shortcoming:

• Computational performance is low and batch parallelization cannot be performed. The main reason is that the number of increased word cells (as nodes) between each character is inconsistent;
• Information loss: 1) Each character can only obtain the lexical information ending with it, and there is no continuous memory for the lexical information before it. For example, for "medicine", the 'inside' information of "Renhe Pharmacy" cannot be obtained. 2) Due to the characteristics of RNN, the forward and backward vocabulary information cannot be shared when BiLSTM is adopted.
• Poor transferability: only suitable for LSTM, does not have the characteristics of migration to other networks