Introduction to Deep Learning (62) Recurrent Neural Network - Bidirectional Recurrent Neural Network

foreword

The core content comes from blog link 1 blog link 2 I hope you can support the author a lot
This article is used for records to prevent forgetting

Recurrent Neural Networks - Bidirectional Recurrent Neural Networks

courseware

the future is important

I am _____
I am _____ very hungry,
I am _____ very hungry, I could eat half a pig.

I am happy.
I am not very hungry,
I am very very hungry, I could eat half a pig.

• Depending on past and future contexts, can fill in very different words
• So far RNN only looks at the past
• While filling in the blanks, we can also see the future

Bidirectional RNN

• — a forward RNN hidden layer
• A backward RNN hidden layer
• Combine the two hidden states to get the output
  

reasoning

Summarize

• Bidirectional Recurrent Neural Networks Utilize Orientational Temporal Information Through Inversely Updated Hidden Layers
• It is usually used to extract features and fill in the blanks of the sequence, not to predict the future

Textbook

In sequence learning, we have historically assumed that the goal is to: Model the next output given an observation (e.g., in the context of a time series or in the context of a language model). While this is a typical scenario, it is not unique. What else could happen? We consider the following three tasks of filling in gaps in text sequences.

I___.
I'm hungry.
I'm ___ hungry, I could eat half a pig.

Depending on the amount of information available, we could fill in the blanks with different words, such as "happy", "not" and "very". Clearly, the "below" of each phrase conveys important information, if any, about which word is chosen to fill in the blank, so sequence models that fail to exploit this will perform poorly on related tasks. For example, to do well in named entity recognition (e.g., identify whether "Green" refers to "Mr. Green" or green), different lengths of context ranges are equally important. To get some inspiration for solving the problem, let's first detour to probabilistic graphical models.

1 Dynamic Programming in Hidden Markov Model

This section is used to illustrate the dynamic programming problem. The specific technical details are not important to understand the deep learning model , but it helps us think about why we use deep learning and why we choose a specific architecture.

If we want to use a probabilistic graphical model to solve this problem, we can design a hidden variable model: at any time step $t$ , assuming there is a hidden variable$h_t$, pass probability $P(x_t\mid h_t)$ controls our observed$x_t$. Furthermore, any $h_t\to h_{t+1}$The transition is determined by some state transition probability $P(h_{t+1}|h_t)$ is given. This probabilistic graphical model is one隐马尔可夫模型（hidden Markov model，HMM）, as shown in the figure.

Therefore, for$For\; a\; sequence\; of\; T$ observations, we have the following joint probability distribution over the observed and hidden states:
$$P(x_1,\dots ,x_T,h_1,\dots ,h_T)=\prod _{t=1}^{T}P(h_t\mid h_{t-1})P(x_t|h_t),\text{ where }P(h_1|h_0)=P(h_1).$$

Now, suppose we observe all $x_i$, except $x_j$, and our goal is to calculate $P(x_j\mid x_{-j})$， 其中$x_{-j}=(x_1,\dots ,x_{j-1},x_{j+1},\dots ,x_T)$ . Since$P(x_j|x_{-j})$ has no hidden variables, so we consider$h_1,\dots ,h_T$All possible combinations formed by the selection are summed. if any $h_i$acceptable $k$ different values ​​(finite number of states), which means we need to know$k^{The\; sum\; of\; T}$ items is obviously a difficult task. Fortunately, there is a neat solution:动态规划（dynamic programming）。

To see how dynamic programming works, we consider the hidden variables $h_1,\dots ,h_T$Summation in turn. According to the above formula, it will be obtained:
$$\begin{array}{r}\begin{array}{rl}& P(x_1,\dots ,x_T)\\ =& \sum _{h_1,\dots ,h_T}P(x_1,\dots ,x_T,h_1,\dots ,h_T)\\ =& \sum _{h_1,\dots ,h_T}\prod _{t=1}^{T}P(h_t\mid h_{t-1})P(x_t|h_t)\\ =& \sum _{h_2,\dots ,h_T}\underset{{Pi}_2(h_2)\stackrel{\mathrm{def}}{=}}{\underbrace{\left[\sum _{h_1}P\left(h_1\right)P(x_1|h_1)P(h_2|h_1)\right]}}P(x_2|h_2\left)\prod _{t=3}^{T}P\right(h_t|h_{t-1})P(x_t|h_t)\\ =& \sum _{h_3,\dots ,h_T}\underset{{Pi}_3\left(h_3\right)\stackrel{def}{=}}{\underbrace{\left[\sum _{h_2}{Pi}_2\left(h_2\right)P(x_2|h_2\left)P\right(h_3|h_2)\right]}}P(x_3|h_3\left)\prod _{t=4}^{T}P\right(h_t|h_{t-1})P(x_t|h_t)\\ =& \dots \\ =& \sum _{h_T}{Pi}_T\left(h_T\right)P(x_T|h_T).\end{array}\end{array}$$

Usually, we will 前向递归（forward recursion）write:
$${Pi}_{t+1}(h_{t+1})=\sum _{h_t}{Pi}_t(h_t)P(x_t\mid h_t)P(h_{t+1}|h_t).$$
The recursion is initialized as${Pi}_1\left(h_1\right)=P(h_1)$ . Symbol simplification, can also be written as${Pi}_{t+1}=f(p_t,x_t)$ , where$f$ is some learnable function. This looks like the update equation in the latent variable model we discussed in Recurrent Neural Networks.

As with forward recursion, we can also use backward recursion to sum the same set of hidden variables. This will get:
$$\begin{array}{r}\begin{array}{rl}& P(x_1,\dots ,x_T)\\ =& \sum _{h_1,\dots ,h_T}P(x_1,\dots ,x_T,h_1,\dots ,h_T)\\ =& \sum _{h_1,\dots ,h_T}\prod _{t=1}^{T-1}P(h_t\mid h_{t-1})P(x_t|h_t)\cdot P(h_T|h_{T-1})P(x_T|h_T)\\ =& \sum _{h_1,\dots ,h_{T-1}}\prod _{t=1}^{T-1}P(h_t|h_{t-1}\left)P\right(x_t|h_t)\cdot \underset{r_{T-1}(h_{T-1})\stackrel{\mathrm{def}}{=}}{\underbrace{\left[\sum _{h_T}P(h_T|h_{T-1}\left)P\right(x_T|h_T)\right]}}\\ =& \sum _{h_1,\dots ,h_{T-2}}\prod _{t=1}^{T-2}P(h_t|h_{t-1}\left)P\right(x_t|h_t)\cdot \underset{r_{T-2}\left(h_{T-2}\right)\stackrel{def}{=}}{\underbrace{\left[\sum _{h_{T-1}}P(h_{T-1}|h_{T-2}\left)P\right(x_{T-1}|h_{T-1})r_{T-1}\left(h_{T-1}\right)\right]}}\\ =& \dots \\ =& \sum _{h_1}P\left(h_1\right)P(x_1|h_1\left)r_1\right(h_1).\end{array}\end{array}$$

Therefore, we can 后向递归（backward recursion）write:
$$r_{t-1}\left(h_{t-1}\right)=\sum _{h_t}P(h_t|h_{t-1}\left)P\right(x_t|h_t)r_t\left(h_t\right),$$
initialize$r_T(h_T)=1$ . Both forward and backward recursion allow us$T$ hidden variables in$O\left(kT\right)$ (linear rather than exponential) time for$(h_1,\dots ,h_T)$ to sum all values. This is one of the great benefits of using graphical models for probabilistic reasoning. It's also消息传递算法a very specific example of a generic. Combining forward and backward recursion, we are able to compute
$$P(x_j\mid x_{-j})\propto \sum _{h_j}{Pi}_j\left(h_j\right)r_j(h_j)P(x_j\mid h_j).$$
Because of symbol simplification, backward recursion can also be written as$r_{t-1}=g(p_t,x_t)$ , where$g$ is a function that can be learned. Again, this looks very much like an update equation, but instead of running forward like we see in recurrent neural networks, it works backwards. In fact, knowing when future data will be available is beneficial for Hidden Markov Models.

2 Two-way model

If we want to have a mechanism in the recurrent neural network that can provide similar forward-looking capabilities as hidden Markov models, we need to modify the design of the recurrent neural network. Fortunately, this is conceptually easy, just adding a RNN that "runs backwards from the last token" instead of just a "starting from the first token" in forward mode running" recurrent neural network. 双向循环神经网络（bidirectional RNNs）Hidden layers that pass information back are added to allow for more flexibility in handling such information. The diagram below depicts the architecture of a bidirectional recurrent neural network with a single hidden layer.

In fact, this is not much different from the forward and backward recursion of dynamic programming in hidden Markov models. The main difference is that the equations in HMMs have specific statistical significance. There is no such easy-to-understand explanation for bidirectional recurrent neural networks, we can only treat them as general-purpose, learnable functions. This shift epitomizes the design principles of modern deep networks: first use the functionally dependent types of classical statistical models, and then parameterize them into a general form.

2.1 Definition

The bidirectional recurrent neural network was proposed by (Schuster and Paliwal, 1997). Let's look at the details of such a network.

For any time step $t$ , given a small batch of input data${\mathbf{X}}_t\in {\mathbb{R}}^{n\times d}$ (number of samples$n$ , the number of inputs ddin each example$d$ ), and let the hidden layer activation function be$\varphi$ . In the bidirectional architecture, we set the forward and reverse hidden states of the time step as${\overrightarrow{\mathbf{H}}}_t\in R^{n\times h}$和 ${\overleftarrow{H}}_t\in R^{n\times h}$ , where$h$ is the number of hidden units. The forward and backward hidden states are updated as follows:
$$\begin{array}{r}\begin{array}{rl}{\overrightarrow{\mathbf{H}}}_t& =\varphi (X_t{\mathbf{W}}_{xh}^{(f)}+{\overrightarrow{H}}_{t-1}{W}_{hh}^{\left(f\right)}+{\mathbf{b}}_h^{\left(f\right)}),\\ {\overleftarrow{H}}_t& =\varphi (X_t{W}_{xh}^{(b)}+{\overleftarrow{H}}_{t+1}{\mathbf{W}}_{hh}^{(b)}+{\mathbf{b}}_h^{\left(b\right)}),\end{array}\end{array}$$
其中，权重$W_{xh}^{(f)}\in {\mathbb{R}}^{d\times h},W_{hh}^{\left(f\right)}\in R^{h\times h},W_{xh}^{\left(b\right)}\in R^{d\times h},W_{hh}^{\left(b\right)}\in R^{h\times h}$和偏置$b_h^{\left(f\right)}\in R^{1\times h},{\mathbf{b}}_h^{(b)}\in {\mathbb{R}}^{1\times h}$ are all model parameters.

Next, the forward hidden state ${\overrightarrow{\mathbf{H}}}_t$and reverse hidden state ${\overleftarrow{H}}_t$Connect them to obtain the hidden state $H_t\in R^{n\times 2h}$ .${}^{\_}$In a deep bidirectional recurrent neural network with multiple hidden layers, this information is passed as input to the next bidirectional layer. Finally, the output calculated by the output layer is${\mathbf{O}}_t\in R^{n\times q}$（$q$ is the number of output units):
$$O_t=H_t{\mathbf{W}}_{hq}+b_q.Here$$
, the weight matrix${\mathbf{W}}_{hq}\in {\mathbb{R}}^{2h\times q}$ and bias${\mathbf{b}}_q\in R^{1\times q}$ is the model parameter of the output layer. In fact, the two directions can have different numbers of hidden units.

2.2 Computational cost of the model and its application

A key property of bidirectional recurrent neural networks is that they use information from both ends of the sequence to estimate the output. That is, we use information from past and future observations to predict current observations. But in the case of predicting the next token, such a model is not what we need. Because when predicting the next token, we can't know what the next token will be after all, so we won't get good accuracy. Specifically, during training, we are able to use past and future data to estimate the present vacant word; while during testing, we only have past data, so the accuracy will be poor. The following experiment will illustrate this point.

Another serious problem is that bidirectional recurrent neural networks are computationally very slow. The main reason is that the forward propagation of the network requires forward and backward recursion in the bidirectional layer, and the backward propagation of the network also depends on the result of the forward propagation. Therefore, the gradient solver will have a very long chain.

The use of bidirectional layers is very rare in practice and is only used in some cases. For example, filling in missing words, lemma annotation (e.g. for named entity recognition), and encoding sequences as a step in a sequence processing pipeline (e.g. for machine translation).

3 Misapplication of bidirectional recurrent neural network

Since bidirectional recurrent neural networks use past and future data, we cannot blindly apply this language model to any prediction task. Although the perplexity levels produced by the model are reasonable, the model's ability to predict future tokens may be severely flawed. We use the sample code below as a warning in case they are used in the wrong context.

import torch
from torch import nn
from d2l import torch as d2l

# 加载数据
batch_size, num_steps, device = 32, 35, d2l.try_gpu()
train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)
# 通过设置“bidirective=True”来定义双向LSTM模型
vocab_size, num_hiddens, num_layers = len(vocab), 256, 2
num_inputs = vocab_size
lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers, bidirectional=True)
model = d2l.RNNModel(lstm_layer, len(vocab))
model = model.to(device)
# 训练模型
num_epochs, lr = 500, 1
d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)

output:

perplexity 1.1, 109857.9 tokens/sec on cuda:0
time travellerererererererererererererererererererererererererer
travellerererererererererererererererererererererererererer

4 Summary

• In the bidirectional recurrent neural network, the hidden state of each time step is determined by the data before and after the current time step at the same time.

• Bidirectional recurrent neural networks have similarities to the "forward-backward" algorithm in probabilistic graphical models.

• Bidirectional recurrent neural networks are mainly used for sequence encoding and estimation of observations given a bidirectional context.

• Bidirectional recurrent neural networks are very expensive to train due to the longer gradient chains.