Attention mechanism (3): Bahdanau attention

Column: Neural Network Recurrence Directory

attention mechanism

Attention Mechanism is an artificial intelligence technology that allows neural networks to focus on key information while ignoring unimportant parts when processing sequence data. The attention mechanism has been widely used in natural language processing, computer vision, speech recognition and other fields.

The main idea of ​​the attention mechanism is to make the model pay more attention to important inputs by assigning different weights to input signals at different positions when processing sequence data. For example, when processing a sentence, the attention mechanism can adjust the model's attention to each word according to the importance of each word. This technique can improve the performance of the model, especially when dealing with long sequence data.

In deep learning models, attention mechanisms are often implemented by adding additional network layers that learn how to compute weights and apply those weights to the input signal. Common attention mechanisms include self-attention, multi-head attention, etc.

In conclusion, the attention mechanism is a very useful technique that can help neural networks to better process sequential data and improve the performance of the model.

---

---

Review of seq2seq

Suppose the input sequence is $x_1,...,x_T$where $x_t$is the ttth in the input text sequence$t$ words. at time step$t$ , the recurrent neural network converts the word unit$x_t$The input feature vector $x_t$and $h_{t-1}$(that is, the hidden state of the previous time step) is converted to $h_t$(i.e. the hidden state of the current step).

Encoder

using a function $f$ to describe the transformation made by the recurrent layer of the recurrent neural network:
$$h_t=f(x_t,h_{t-1})$$
In summary, the encoder passes the selected function$q$ , convert the hidden state of all time steps into context variables:
$$c=q(h_1,...,h_T)$$

decoder

The context variable cc output by the encoder$c$ for the entire input sequence$x_1,...,x_T$to encode. Output sequence y 1 , . . . , y T y_1,...,y_T from the training dataset$y_1,...,y_T$, for each time step $t^{\prime }$ (with input sequence or encoder time step$t$ different), the decoder outputs$y^{\prime }$ depends on the previous output subsequence$y_1,...,y_{t-1}$and the context variable $c$， 即$P({and}_t^{\prime }|y_1,..,y_{t^{\prime }-1},c)$。

To model this conditional probability on sequences, we can use another recurrent neural network as a decoder. At any time step t' t' on the output sequence$t^{\prime }$ , the RNN will output$y_{t^{\prime }-1}$and the context variable $c$ as its input, and then combine them with the previous hidden state$s_{t^{\prime }-1}$Convert to hidden state $s_{t^{\prime }}$. Therefore, the function $g$ to represent the transformation of the hidden layer of the decoder:
$$s_{t^{\prime }}=g(y_{t^{\prime }-1},c,s_{t^{\prime }-1})$$

attention

Bahdanau attention is an attention mechanism used in neural machine translation, first proposed by Dzmitry Bahdanau et al. in 2015.

In traditional neural machine translation models, an encoder encodes a source language sentence into a fixed-length vector, and a decoder uses this vector to generate a target language sentence. However, this method of fixed-length vectors has some disadvantages, such as the inability to handle longer sentences and the inability to capture complex dependencies between different words in a sentence.

Bahdanau attention addresses these issues by introducing a dynamic alignment mechanism. At each time step, the decoder computes an alignment distribution based on the previous output and all hidden states of the encoder, which indicates which positions in the encoder are most useful for the output at the current time step. The decoder then uses the weighted encoder hidden states to generate the output at the current time step.

This dynamic alignment mechanism can enable the model to better capture the complex dependencies between the source language and the target language, thereby improving the accuracy and fluency of translation. Bahdanau attention has been widely used in machine translation, speech recognition, image annotation and other fields, and has become one of the mainstream methods in the field of neural machine translation.

Bahdanau model

This new attention-based model is the same as the one above, except for the context variable $c$ at any decoding time step$t^{\prime }$ will be$c_{t^{\prime }}$replace. Suppose there is $T$ tokens, decoding time step$t^{\prime }$ is the output of attention focus:
$$c_{t^{\prime }}=\sum _{t=1}^{T}a(s_{t^{\prime }-1},h_t)h_t$$
where, time step $t^{\prime }-$The hidden state of the decoder at$1$$s_{t^{\prime }-1}$is the query, the encoder hidden state $h_t$It is both a key and a value, and the attention weight $\alpha$ is computed using an additive attention scoring function.

accomplish

model definition

#@save
class AttentionDecoder(d2l.Decoder):
    """带有注意力机制解码器的基本接口"""
    def __init__(self, **kwargs):
        super(AttentionDecoder, self).__init__(**kwargs)

    @property
    def attention_weights(self):
        raise NotImplementedError

Next, let's implement a recurrent neural network decoder with Bahdanau attention in the next Seq2SeqAttentionDecoder class. First, initialize the state of the decoder, requiring the following inputs:

1. The hidden state of the final layer of the encoder at all time steps will be used as the key and value of attention;

2. The hidden state of the full layer of the encoder at the previous time step will be used as the hidden state of the initialized decoder;

3. Encoder effective length (excluding padding tokens in the attention pool).

At each decoding time step, the final layer hidden state from the previous time step of the decoder is used as a query. Therefore, both the attention output and the input embedding are concatenated as input to the RNN decoder.

class Seq2SeqAttentionDecoder(AttentionDecoder):
    def __init__(self, vocab_size, embed_size, num_hiddens, num_layers,
                 dropout=0, **kwargs):
        super(Seq2SeqAttentionDecoder, self).__init__(**kwargs)
        self.attention = d2l.AdditiveAttention(
            num_hiddens, num_hiddens, num_hiddens, dropout)
        self.embedding = nn.Embedding(vocab_size, embed_size)
        self.rnn = nn.GRU(
            embed_size + num_hiddens, num_hiddens, num_layers,
            dropout=dropout)
        self.dense = nn.Linear(num_hiddens, vocab_size)

    def init_state(self, enc_outputs, enc_valid_lens, *args):
        # outputs的形状为(batch_size，num_steps，num_hiddens).
        # hidden_state的形状为(num_layers，batch_size，num_hiddens)
        outputs, hidden_state = enc_outputs
        return (outputs.permute(1, 0, 2), hidden_state, enc_valid_lens)

    def forward(self, X, state):
        # enc_outputs的形状为(batch_size,num_steps,num_hiddens).
        # hidden_state的形状为(num_layers,batch_size,
        # num_hiddens)
        enc_outputs, hidden_state, enc_valid_lens = state
        # 输出X的形状为(num_steps,batch_size,embed_size)
        X = self.embedding(X).permute(1, 0, 2)
        outputs, self._attention_weights = [], []
        for x in X:
            # query的形状为(batch_size,1,num_hiddens)
            query = torch.unsqueeze(hidden_state[-1], dim=1)
            # context的形状为(batch_size,1,num_hiddens)
            context = self.attention(
                query, enc_outputs, enc_outputs, enc_valid_lens)
            # 在特征维度上连结
            x = torch.cat((context, torch.unsqueeze(x, dim=1)), dim=-1)
            # 将x变形为(1,batch_size,embed_size+num_hiddens)
            out, hidden_state = self.rnn(x.permute(1, 0, 2), hidden_state)
            outputs.append(out)
            self._attention_weights.append(self.attention.attention_weights)
        # 全连接层变换后，outputs的形状为
        # (num_steps,batch_size,vocab_size)
        outputs = self.dense(torch.cat(outputs, dim=0))
        return outputs.permute(1, 0, 2), [enc_outputs, hidden_state,
                                          enc_valid_lens]

    @property
    def attention_weights(self):
        return self._attention_weights

This code implements the core principles of Bahdanau's attention model. The Bahdanau attention model is a model for sequence-to-sequence (seq2seq) learning. Its core idea is to dynamically select the output of the encoder according to the current decoder state and the output of the encoder at each step of the decoder. different parts to calculate the output at the current moment.

Specifically, the implementation process of this code segment is as follows:

1. At each step of the decoder, first feed the current decoder into $x$ and encoder$enc\_outputs$ perform attention calculations to obtain the current context vector$context$ .$\_$The attention calculation uses the Additive Attention mechanism, where$query$ indicates the state of the decoder at the current moment,$keys$ 和 $values$ are the output of the encoder$enc\_outputs$，$enc\_valid\_lensis$ the effective length of the encoder.

2. Input $x$ and the context vector$Contextis$ spliced ​​on the feature dimension to get$a$$new$$input$$x$。

3. Enter the new $x$ and the decoder state hidden_state$hidden\_state$ is input to GRU, and the output at the current moment$out$ and new decoder state$hidden\_state$。

4. Output the output at the current moment $out$ added to the output list$outputs$ 中。

5. Add the attention weight at the current moment to the attention weight list _ attention _ weights \_attention\_weightsDuring$\_$$a$$tt$$e$$n$$t$$i$$o$$n$$\_$$w$$e$$i$$g$$h$$t$$s\; .$

Repeat the above steps until all moments of the decoder output sequence are calculated.

During the implementation process, it should be noted that the dimensions of the input data need to be consistent with the design of the model. Specifically, the input data $The\; dimension\; of\; X$ should be$(batch\_size,seq\_length)$，其中 $batc{h}_{s}ize$ indicates the batch size,$se{q}_{l}ength$ represents the sequence length. The input required by the recurrent neural network (RNN) layer in the model is$(seq\_length,batch\_size,input\_size)$ , so it is necessary to perform dimension transformation on the input data. In this code, the permute() function in PyTorch is used to transform the dimensions of the input data.

At the same time, when calculating the attention at each step, it is also necessary to enc_valid_lens according to the effective length of the encoder $en{c}_{v}ali{d}_{l}ens$ performs Mask processing to avoid the influence of invalid positions on the attention calculation$.$Here, the call() function in the Additive Attention class is used to automatically mask the invalid positions.

Specifically, the decoder's constructor accepts the following parameters:

vocab_size: the size of the vocabulary
embed_size: the size of the embedding vector
num_hiddens: the number of hidden units in the RNN layer
num_layers: the number of layers in the RNN layer
dropout: the probability of dropout

In the constructor, the decoder first calls the constructor of its parent class AttentionDecoder for initialization. Then, it defines three modules:

self.attention: An additive attention model that takes as input the output of the encoder and the hidden state of the decoder, and returns a context vector to guide the decoder to generate the target sequence at the current time step.

self.embedding: An embedding layer that converts the target sequence into an embedding vector.

self.rnn: An RNN layer with a GRU unit that concatenates the context vector and embedding vector and returns the output and hidden state at the current time step.

Next, the decoder defines an init_state method that takes the encoder's output, the encoder's effective length, and any other parameters and transforms them into the decoder's initial state. This method swaps the first and second dimensions of the encoder's output and adds the effective length as a third element to the state.

Finally, the decoder defines a forward method that takes as input the target sequence and the state of the decoder, and returns the resulting sequence and its final state. In this method, the target sequence is first converted into an embedding vector, and then each time step in the sequence is iterated. At each time step, the decoder uses an attention mechanism to compute a context vector and concatenates it with the embedding vector for input into the RNN. The decoder then adds the RNN output to one list and the attention weights to another list. Finally, the decoder uses a fully-connected layer to transform all outputs into a probability distribution, and returns the resulting sequence and its state. Attention weights can be accessed through the property self.attention_weights.

In natural language processing, word embedding (Word embedding) refers to mapping words into a low-dimensional real number vector space. By representing each word as a vector, it can be processed more conveniently in a neural network.
In this code, nn.Embedding is a PyTorch module used to define a word vector embedding layer. In the initialization function of Seq2SeqAttentionDecoder, self.embedding is an instance of nn.Embedding, which is used to encode words into corresponding word vectors. Specifically, given an integer, which corresponds to the number of a word, the embedding layer converts the integer into a fixed-size vector, which is used as input in the neural network.
In this code, self.embedding receives X as input, where X is an integer tensor with shape (batch_size, num_steps). This embedding layer converts each integer into a word vector, and these word vectors form a tensor of shape (batch_size, num_steps, embed_size). In this way, we can encode a sequence of text into a corresponding sequence of word vectors and use these vectors in a neural network for processing.

encoder = d2l.Seq2SeqEncoder(vocab_size=10, embed_size=8, num_hiddens=16,
                             num_layers=2)
encoder.eval()
decoder = Seq2SeqAttentionDecoder(vocab_size=10, embed_size=8, num_hiddens=16,
                                  num_layers=2)
decoder.eval()
X = torch.zeros((4, 7), dtype=torch.long)  # (batch_size,num_steps)
state = decoder.init_state(encoder(X), None)
output, state = decoder(X, state)
output.shape, len(state), state[0].shape, len(state[1]), state[1][0].shape

train

embed_size, num_hiddens, num_layers, dropout = 32, 32, 2, 0.1
batch_size, num_steps = 64, 10
lr, num_epochs, device = 0.005, 250, d2l.try_gpu()

train_iter, src_vocab, tgt_vocab = d2l.load_data_nmt(batch_size, num_steps)
encoder = d2l.Seq2SeqEncoder(
    len(src_vocab), embed_size, num_hiddens, num_layers, dropout)
decoder = Seq2SeqAttentionDecoder(
    len(tgt_vocab), embed_size, num_hiddens, num_layers, dropout)
net = d2l.EncoderDecoder(encoder, decoder)
d2l.train_seq2seq(net, train_iter, lr, num_epochs, tgt_vocab, device)

test

engs = ['go .', "i lost .", 'he\'s calm .', 'i\'m home .']
fras = ['va !', 'j\'ai perdu .', 'il est calme .', 'je suis chez moi .']
for eng, fra in zip(engs, fras):
    translation, dec_attention_weight_seq = d2l.predict_seq2seq(
        net, eng, src_vocab, tgt_vocab, num_steps, device, True)
    print(f'{
      
      eng} => {
      
      translation}, ',
          f'bleu {
      
      d2l.bleu(translation, fra, k=2):.3f}')

attention visualization

# 加上一个包含序列结束词元
d2l.show_heatmaps(
    attention_weights[:, :, :, :len(engs[-1].split()) + 1].cpu(),
    xlabel='Key positions', ylabel='Query positions')