Bidirectional Recurrent Neural Network

Bidirectional RNN structure model diagram

A forward RNN hidden layer and a backward RNN hidden layer combine the two hidden states to the output.

formula expression

Application example (cloze)

The first line is the normal RNN prediction, the last two lines can use bidirectional RNN for word filling, and the result depends on the past and future context.

pros and cons

As shown in the figure below, bidirectional RNNs are suitable for training because they provide past and future information during training.

However, for inference, i.e. prediction, a bidirectional RNN cannot be implemented because it requires not only past information, but also future information. However, the information in the future is exactly the information we want to reason about, so it cannot be realized.

Summarize

According to the above training process, we can extract the features of the text through the bidirectional RNN , so that the extracted features are related to the context, and are often used for sentence translation.