Introduction to Deep Learning (62) Recurrent Neural Network - Deep Recurrent Neural Network

foreword

The core content comes from blog link 1 blog link 2 I hope you can support the author a lot
This article is used for records to prevent forgetting

Recurrent Neural Networks - Deep Recurrent Neural Networks

courseware

Recap: Recurrent Neural Networks

Update hidden state: $h_t=\varphi (W_{hh}{h}_{t-1}+W_{hx}{x}_{t-1})+b_h$
Export: $o_t=\varphi (W_h{h}_t+b_o)$

How to get more non-linearity?

PLANA：Nonlinearity in the units

$h_t=\varphi (W_{hh}{h}_{t-1}+W_{hx}{x}_{t-1})+b_h$
$o_t=\varphi (W_h{h}_t+b_o)$

Replace $\varphi$

Deeper

Summarize

• Deep Recurrent Neural Networks use multiple hidden layers for more nonlinearity

courseware

So far, we have only discussed recurrent neural networks with one unidirectional hidden layer. The way in which hidden variables and observations interact with the specific functional form is quite arbitrary. This is not a huge problem as long as there is sufficient flexibility in modeling the interaction types. However, this can be quite challenging for a single layer. Previously in linear models we solved this problem by adding more layers. And in recurrent neural networks, we first need to figure out how to add more layers, and where to add the extra non-linearity, so this problem is a bit trickier.

In fact, we can stack multiple layers of recurrent neural networks together, resulting in a flexible mechanism through the composition of a few simple layers. In particular, data may relate to stacks of different layers. For example, we may wish to keep macro data on financial market conditions (bear or bull) available, while micro data only capture shorter-term temporal dynamics.

The figure below depicts a deep recurrent neural network with N hidden layers, each hidden state is passed continuously to the next time step of the current layer and the current time step of the next layer.

1 Functional dependencies

We can formalize the functional dependencies in the deep architecture, which is described by $It\; consists\; of\; L$ hidden layers. Subsequent discussions focus on classical recurrent neural network models, but these discussions are also applicable to other sequence models.

Suppose at time step $t$ has a small batch of input data${\mathbf{X}}_t\in {\mathbb{R}}^{n\times d}$ (number of samples:$n$ , the number of inputs in each sample:$d$ ). At the same time,$l^{th}$ hidden layer ($l=1,\dots ,L$）The hidden state is set to${\mathbf{H}}_t^{(l)}\in {\mathbb{R}}^{n\times h}$ (number of hidden units:$h$ ), the output layer variable is set to${\mathbf{O}}_t\in R^{n\times q}$ (Number of outputs:$q$）。 设置$H_t^{(0)}={\mathbf{X}}_t$, th $The\; hidden\; state\; of\; the\; l$ hidden layer uses the activation function${\varphi }_l$，则：
$${\mathbf{H}}_t^{(l)}={\varphi }_l({\mathbf{H}}_t^{(l-1)}{\mathbf{W}}_{xh}^{\left(l\right)}+H_{t-1}^{\left(l\right)}{W}_{hh}^{\left(l\right)}+{\mathbf{b}}_h^{\left(l\right)}),$$
where, weight$W_{xh}^{\left(l\right)}\in {\mathbb{R}}^{h\times h}$,$W_{hh}^{\left(l\right)}\in R^{h\times h}$ and bias${\mathbf{b}}_h^{(l)}\in {\mathbb{R}}^{1\times h}$ is the$The\; model\; parameters\; of\; the\; l$ hidden layers.

Finally, the calculation of the output layer is based only on the llthThe final hidden state of$l$
$${\mathbf{O}}_t={\mathbf{H}}_t^{(L)}{\mathbf{W}}_{hq}+b_q,$$
where, the weight$W_{hq}\in R^{h\times q}$ and bias$b_q\in R^{1\times q}$ are the model parameters of the output layer.

Like the multi-layer perceptron, the number of hidden layers $L$ and the number of hidden units$h$ are hyperparameters. That is, they can be adjusted by us. Also, replace${\mathbf{H}}_t^{(l)}={\varphi }_l({\mathbf{H}}_t^{(l-1)}{\mathbf{W}}_{xh}^{\left(l\right)}+H_{t-1}^{\left(l\right)}{W}_{hh}^{\left(l\right)}+{\mathbf{b}}_h^{\left(l\right)}),$ the hidden state in , can be easily obtained deep gated recurrent neural network or deep long short-term memory neural network.

2 Concise implementation

Many of the logical details needed to implement multilayer recurrent neural networks are readily available in the high-level API. For simplicity, we only demonstrate implementations using such built-in functions. Taking the LSTM network model as an example, the code is very similar to the code we used earlier in the LSTM section, the only difference in fact is that we specify the number of layers instead of using the default value of a single layer. As usual, we start by loading the dataset.

import torch
from torch import nn
from d2l import torch as d2l

batch_size, num_steps = 32, 35
train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)

Architectural decisions like choosing hyperparameters are also very similar to those in the LSTM section. Since we have different tokens, the same number is chosen for both input and output, ie vocab_size. The number of hidden units is still 256. The only difference is that we now pass num_layersthe value to set the number of hidden layers.

vocab_size, num_hiddens, num_layers = len(vocab), 256, 2
num_inputs = vocab_size
device = d2l.try_gpu()
lstm_layer = nn.LSTM(num_inputs, num_hiddens, num_layers)
model = d2l.RNNModel(lstm_layer, len(vocab))
model = model.to(device)

3 Training and Prediction

Since the LSTM network model is used to instantiate two layers, the training speed is greatly reduced.

num_epochs, lr = 500, 2
d2l.train_ch8(model, train_iter, vocab, lr*1.0, num_epochs, device)

output:

perplexity 1.0, 224250.2 tokens/sec on cuda:0
time travelleryou can show black is white by argument said filby
travelleryou can show black is white by argument said filby

4 Summary

• In deep recurrent neural networks, the information of the hidden state is passed to the next time step of the current layer and the current time step of the next layer.

• There are many different flavors of deep recurrent neural networks, such as long short-term memory networks, gated recurrent units, or classical recurrent neural networks. These models are covered in the high-level APIs of deep learning frameworks.

• Overall, deep recurrent neural networks require extensive tuning of parameters (such as learning rate and pruning) to ensure proper convergence, and model initialization also requires care.