Artificial intelligence (pytorch) building model 8 - use pytorch to build a BiLSTM+CRF model to achieve simple named entity recognition

Hello everyone, I am Weixue AI. Today I will introduce to you artificial intelligence (pytorch) building model 8-Use pytorch to build a BiLSTM+CRF model to achieve simple named entity recognition. The BiLSTM+CRF model is a commonly used sequence annotation Algorithms that can be used for tasks such as part-of-speech tagging, word segmentation, and named entity recognition. This article uses pytorch to build a BiLSTM+CRF model, and gives data samples, and demonstrates the training and prediction process of the model through a simple named entity recognition (NER) task. The article will be divided into the following sections:

1. Introduction to the BiLSTM+CRF model2
. Mathematical principles of the BiLSTM+CRF model3
. Data preparation4
. Model building5
. Training and evaluation6
. Prediction7
. Summary

1. Introduction to the BiLSTM+CRF model

The BiLSTM+CRF model combines two technologies of bidirectional long short-term memory network (BiLSTM) and conditional random field (CRF). BiLSTM is used to capture contextual information in sequences, while CRF is used to resolve dependencies between labels. In fact, BiLSTM is used to generate a feature vector for each input sequence, and then these feature vectors are fed into the CRF layer to assign a label to each element in the sequence. The combination of BiLSTM and CRF makes the model not only consider the correlation between the sequence before and after the sequence like CRF, but also have the feature extraction and fitting capabilities of LSTM.

2. Mathematical principle of BiLSTM+CRF model

Suppose we have a sequence $\mathbf{x}=(x_1,x_2,...,x_n)$ , of which$x_i$right $Input\; features\; for\; i$ positions. We want to label each position, that is, for each position$i$ predicts a label$y_i$. The label set is $\mathcal{Y}=y_1,y_2,...,y_n$, where $y_i\in L$，$L$ represents the category set of labels.

BiLSTM is used to extract features from the input sequence, which consists of LSTMs in two directions, processing the input sequence from front to back and from back to front, respectively. at time step $t$ , the output of BiLSTM is$h_t\in {\mathbb{R}}^{2d}$ , where$d$ is the hidden state dimension of LSTM. Specifically, the forward LSTM processes the input sequence$x$ , output hidden state sequence$\overrightarrow{h}=(\overrightarrow{h_1},\overrightarrow{h_2},...,\overrightarrow{h_n})$ , where$\overrightarrow{h_t}$Indicates that at time step ttThe hidden state of the forward LSTM at$t$$x$ , output hidden state sequence$\overleftarrow{h}=(\overleftarrow{h_1},\overleftarrow{h_2},...,\overleftarrow{h_n})$ , where$\overleftarrow{h_t}$Indicates that at time step ttThe hidden state of the backward LSTM at$t\; .$Then each position$The\; feature\; of\; i$ is expressed as$h_i=[\overrightarrow{h_i};\overleftarrow{h_i}]$ , where$[\cdot ;\cdot ]$ represents the vector concatenation operation.

CRF is used to model the relationship between labels and perform global optimization. The CRF model defines a Y \mathcal{Y}The joint distribution P ( y ∣ x ) composed of$Y$$P(y|x)$，其中$\mathbf{y}=(y_1,y_2,...,y_n)$ represents a label sequence. Specifically, the CRF model decomposes the probability of a label sequence into the product of conditional probabilities at multiple positions, namely

$$P(\mathbf{y}|\mathbf{x})=\prod _{i=1}^n{p}_i(y_i|\mathbf{x})\prod _{i=1}^{n-1}{p}_{i,i+1}(y_i,y_{i+1}|x)$$

其中$p_i(y_i|x)$ means at position$When\; i$ , the predicted label is$y_i$, $p_{i,i+1}(y_i,y_{i+1}|x)$ indicates that the predicted label is$y_i$Sum $y_{i+1}$the joint probability of . These conditional and joint probabilities can be modeled with a neural network, where the input is position $The\; feature\; of\; i$ represents$h_i$。

The global optimization problem of the CRF model can be realized by maximizing the log-likelihood function, namely

$$\underset{\mathbf{y}}{\max }\log P(\mathbf{y}|\mathbf{x})=\sum _{i=1}^n\log p_i(y_i|\mathbf{x})\sum _{i=1}^{n-1}\log p_{i,i+1}(y_i,y_{i+1}|x)$$
where$y$ is all possible label sequences. A dynamic programming algorithm such as the Viterbi algorithm can be used to solve for the globally optimal label sequence.

In summary, the mathematical principle of the BiLSTM+CRF model can be expressed as:

$$P(\mathbf{y}|\mathbf{x})=\prod _{i=1}^n{p}_i(y_i|\mathbf{x})\prod _{i=1}^{n-1}{p}_{i,i+1}(y_i,y_{i+1}|x)$$

in

$$p_i(y_i|\mathbf{x})=\frac{\exp (W_o^T{h}_i+b_o^{T}yi}{\sum y_i^{\prime }\in \mathcal{L}\exp ({\mathbf{W}}_o^T{\mathbf{h}}_i+{\mathbf{b}}_o^T{y}_i^{\prime })}$$

$$p_{i,i+1}(y_i,y_{i+1}|\mathbf{x})=\frac{\exp (W{t}^{T}yi,i+1)}{{\sum }_{y_i^{\prime }\in \mathcal{L}}{\sum }_{y_{i+1}^{\prime }\in L}\exp (\mathbf{W}{t}^{T}y{i^{\prime },i+1}^{\prime })}$$

Among them ${\mathbf{W}}_o$和${\mathbf{b}}_o$is the parameter of the output layer, $W_t$is the transition matrix, ${\mathbf{h}}_i$is position $The\; feature\; representation\; of\; i$ ,$yi$is position$The\; label\; of\; i$ means,$yesyes\_i+1$ is position$i$ sum$i+1$ 's label joint representation.

3. Data preparation

Below I will use a simple Named Entity Recognition (NER) task to demonstrate the model training and prediction process. The dataset consists of sentences, and the words in each sentence are labeled as "B-PER" (beginning of person name), "I-PER" (middle of person name), "B-LOC" (beginning of place name), "I- LOC" (place name middle) or "O" (other).

Data sample:

John B-PER
lives O
in O
New B-LOC
York I-LOC
. O

4. Model building

First, we need to install the PyTorch library:

pip install torch

Next, we will use PyTorch to build the BiLSTM+CRF model. The complete model code is as follows:

import torch
import torch.nn as nn
import torch.optim as optim

from TorchCRF import CRF

class BiLSTM_CRF(nn.Module):
    def __init__(self, vocab_size, tag_to_ix, embedding_dim, hidden_dim):
        super(BiLSTM_CRF, self).__init__()
        self.embedding_dim = embedding_dim
        self.hidden_dim = hidden_dim
        self.vocab_size = vocab_size
        self.tag_to_ix = tag_to_ix
        self.tagset_size = len(tag_to_ix)

        self.word_embeds = nn.Embedding(vocab_size, embedding_dim)
        self.lstm = nn.LSTM(embedding_dim, hidden_dim // 2,
                            num_layers=1, bidirectional=True)

        self.hidden2tag = nn.Linear(hidden_dim, self.tagset_size)
        self.crf = CRF(self.tagset_size)

    def forward(self, sentence):
        embeds = self.word_embeds(sentence).view(len(sentence), 1, -1)
        lstm_out, _ = self.lstm(embeds)
        lstm_out = lstm_out.view(len(sentence), self.hidden_dim)
        lstm_feats = self.hidden2tag(lstm_out)
        return lstm_feats

    def loss(self, sentence, tags):
        feats = self.forward(sentence)
        return -self.crf(torch.unsqueeze(feats, 0), tags)

    def predict(self, sentence):
        feats = self.forward(sentence)
        return self.crf.decode(torch.unsqueeze(feats, 0))

5. Training and Evaluation

Next, we will train the model using the training data and print the loss and accuracy after each epoch.

def train(model, optimizer, data):
    for epoch in range(10):
        total_loss = 0
        total_correct = 0
        total_count = 0
        for sentence, tags in data:
            model.zero_grad()
            loss = model.loss(sentence, tags)
            loss.backward()
            optimizer.step()
            total_loss += loss.item()

            prediction = model.predict(sentence)
            total_correct += sum([1 for p, t in zip(prediction, tags) if p == t])
            total_count += len(tags)

        print(f"Epoch {
      
      epoch + 1}: Loss = {
      
      total_loss / len(data)}, Accuracy = {
      
      total_correct / total_count}")

6. Forecast

Finally, we will use the trained model to make predictions on new sentences.

def predict(model, sentence):
    prediction = model.predict(sentence)
    return [p for p in prediction]

7. Summary

Use the trained model to predict new sentences.

def predict(model, sentence):
    prediction = model.predict(sentence)
    return [p for p in prediction]

7. Summary

This article introduces how to use PyTorch to build a BiLSTM+CRF model, and demonstrates the training and prediction process of the model through a simple named entity recognition (NER) task. I hope this article can help you understand the principle of the BiLSTM+CRF model, and provide a reference for your actual project.

Please continue to pay attention to the updated and exciting model building and application!