Neural Network Study Notes 9 - LSTM and GRU Model Understanding and Code Analysis in Recurrent Neural Networks

Series Article Directory

LSTM Video Reference

GRU Video Reference

---

---

foreword

Recurrent Neural Network (RNN) is a neural network for processing sequence data. Compared with the general neural network, it can handle the data of sequence change. For example, the meaning of a word will have different meanings due to the different content mentioned above, and RNN can solve such problems well.
LSTM is a kind of RNN, which can solve the problem of gradient disappearance and gradient explosion in the long sequence training process of RNN. When a sequence is long enough, it will be difficult for RNN to transfer information from earlier time steps to later time steps. LSTM can learn long-term dependent information and remember information from earlier time steps, so it can be associated with context.
Different from RNN, RNN wants to remember all the information, and does not care whether it is useful or not. LSTM designs a memory cell, which functions as a screening function and has the function of selecting memory, which is used to select and remember important information. , to filter noise and non-important information to reduce memory load. Its appearance solves the problem of gradient distortion. Moreover, the convergence speed of RNN is much faster than that of ordinary RNN.

The core concept of LSTM is cell state and "gate" structure. The cell state is equivalent to the path of information transmission, so that information can be passed on in the serial connection. You can think of it as the "memory" of the network. Theoretically, the state of the cell can carry relevant information all the way through the sequence processing.

GRU (Gate Recurrent Unit) is a kind of recurrent neural network (RNN), which can solve the problems of RNN that cannot be long-term memory and gradient in backpropagation. It is similar to LSTM, but it is simpler than LSTM and easy to train. It is very similar to LSTM. Compared with LSTM, GRU removes the cell state and uses hidden state to transmit information. It contains only two gates: the update gate and the reset gate.

GRU is a variant of LSTM, and it is also proposed to solve problems such as long-term memory and gradient in backpropagation. In many cases, GRU and LSTM have almost the same performance, but GRU calculation is simpler and easier to implement.

1. LSTM model structure

RNN structure

LSTM structure

The LSTM model is compared with the RNN model:

1. $X_t$The complexity of the model structure of the position varies greatly
2. LSTM has two outputs, RNN has only one output

Put $X_t$Take it out separately, where
$C$ means memory cell, the formula is Cell State
$h$ means state
$X$ means new content
$\sigma$ represents the gate unit
$f_t$Forget Gate, the formula is Forget Gate
$i_t$Input gate, the formula is Input Gate
$o_t$Output gate, the formula is Output Gate

vs. $f_t$，$i_t$，$o_t$It can be found that their laws are $s(W\times X_t+W\times h_{t-1}+b)$ type. According to the above model structure diagram, it can be found that$(X_t,h{}_t-1)$ will be input into the σ gate, so the X and h values ​​of the three gates are the same. Through$(X_t,h{}_t-1)$ Four values ​​ft f_tcan be calculated$f_t$，$i_t$，$\tilde{C}$and $o_t$. where $tanh$ is the activation function and each element of its output vector is between -1~1,$W,b$ is the parameter of each gate neuron, which is to be learned during the training process.

1. Forget Gate : Decide what information should be discarded or kept. The information from the previous hidden state and the current input information are passed to the sigmoid function at the same time, and the output value is between 0 and 1. The closer to 0 means that it should be discarded, and the closer to 1 means that it should be retained. ft f_t will be obtained$f_t$with input $C_{t-1}$Multiply, that is, vector multiplication.
2. Input Gates : Input gates are used to update the cell state. First, the information of the hidden state of the previous layer and the information of the current input are passed to the sigmoid function. Adjust the value between 0~1 to decide which information to update. 0 means unimportant, 1 means important, get $i_t$value. Secondly, the information of the hidden state of the previous layer and the information of the current input must be passed to the tanh function to create a new candidate value vector and get $\tilde{C}$value. Finally, the output value of sigmoid and the output value of tanh $i_t$and $\tilde{C}$Multiplication, that is, vector multiplication, where the output value of sigmoid will determine which information in the output value of tanh is important and needs to be preserved. The information to be retained will be added to the output obtained by the forget gate, and finally a new memory cell $C_t=(f_t⨀C_{t-1})+(i_t⨀\tilde{C})$
3. Output gate : The output gate is used to determine the value of the next hidden state, which contains the information of the previous input. First, we pass the previous hidden state and the current input into the sigmoid function, and then pass the newly obtained cell state into the tanh function. Finally, the output of tanh is multiplied by the output of sigmoid, that is, vector multiplication, to determine the information that the hidden state should carry. Then use the hidden state as the output of the current cell, and use the new cell state and the new hidden state $h_t$passed to the next time step. That is, the newly obtained memory cells $C_t$Pass in tanh for weight calculation $m_t=I\left(older\right){}_t)$ , then$o_t$According to the weight $m_t$Compute the new hidden state $h_t=o_t⨀m_t$, so as to control which part needs to be output and which part is the answer we need.

2. GRU model structure

Formula expression:

$z_t=s(W_{xz}\times X_t+W_{hz}\times h_{t-1}+b_z)$
$r_t=s(W_{xr}\times X_t+W_{hr}\times h_{t-1}+b_r)$
${\tilde{h}}_t=English(W\_{}_{xh}\times X_t+W_{hh}\times (r_t⨀h_{t-1})+b_h)$
$h_t=(1-z_t)⨀h_{t-1}+z_t⨀{\tilde{h}}_t$

GRU has only two gates. The combination of the input gate and the forget gate in LSTM is called the update gate. The formula is $z_t$, while another formula called reset gate is $r_t$.
It can be observed that the formulas of the two gates are also $s(W\times X_t+W\times h_{t-1}+b)$ type, but here${\tilde{h}}_t$and LSTM's $\tilde{C}$Different, LSTM's $\tilde{C}$is by $h_{t-1},X_t$Composed of, while GRU's ${\tilde{h}}_t$is given by $（r_t⨀h_{t-1}),X_t$consist of. where $tanh$ is the activation function and each element of its output vector is between -1~1,$W,b$ is the parameter of each gate neuron, which is to be learned during the training process.

1. update gate $z_t$: A mechanism that plays a role of attention, used to control the degree to which the state information of the previous moment is brought into the current state, that is, the update gate helps the model decide how much past information to pass to the future, in simple terms, it uses to update the memory.
2. reset gate $r_t$: The mechanism that plays a role in forgetting, determines how to combine new input information with previous memories, controls how much past information to forget, makes the hidden state forget any information that is not related to predictions in the future, and also allows the construction of Tighter representation.

3. Comparison between GRU and LSTM

advantage:

1. Compared with LSTM, GRU has fewer output gates, its parameters are less than LSTM, and the amount of calculation is less than LSTM. It is simpler, easier to implement, and can train the model faster;
2. GRU has only two gates, LSTM has three gates, GRU is easier to control parameters and easier to adjust;

shortcoming:

1. GRU does not have as strong a memory capacity as LSTM, and for long-term dependency problems, LSTM is strictly stronger than GRU because it can easily count infinitely, while GRU cannot. This is why GRUs cannot learn simple languages, which LSTMs can learn;
2. GRU does not have as many parameters as LSTM, and it is prone to underfitting.

4. Code implementation

1. GRU code

From hands-on deep learning

import torch
from torch import nn
from d2l import torch as d2l

batch_size, num_steps = 32, 35
train_iter, vocab = d2l.load_data_time_machine(batch_size, num_steps)

def get_params(vocab_size, num_hiddens, device):
    num_inputs = num_outputs = vocab_size

    def normal(shape):
        return torch.randn(size=shape, device=device)*0.01

    def three():
        # torch.normal：返回一个张量，包含从给定参数num_inputs, num_hiddens的离散正态分布中抽取随机数
        # torch.zeros：创建num_hiddens大小的维度，里面元素全部填充为0device输出张量的设备
        return (normal((num_inputs, num_hiddens)),
                normal((num_hiddens, num_hiddens)),
                torch.zeros(num_hiddens, device=device))

    # 初始化模型参数，定义更新门z_t、重置门r_t和隐藏态h_t公式的训练参数
    # 通过def three()给这些参数定义随机数和和0矩阵
    W_xz, W_hz, b_z = three()  # 更新门参数
    W_xr, W_hr, b_r = three()  # 重置门参数
    W_xh, W_hh, b_h = three()  # 候选隐状态参数

    # 输出层参数
    W_hq = normal((num_hiddens, num_outputs))
    b_q = torch.zeros(num_outputs, device=device)

    # requires_grad：附加梯度，设置参数为True，程序将会追踪所有对于该张量的操作
    # 当完成计算后通过调用 .backward()，自动计算所有的梯度， 这个张量的所有梯度将会自动积累到 .grad 属性。
    params = [W_xz, W_hz, b_z, W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q]
    for param in params:
        param.requires_grad_(True)
    return params

# 定义隐状态的初始化函数，返回一个形状为（批量大小，隐藏单元个数）的张量，张量的值全部为零
def init_gru_state(batch_size, num_hiddens, device):
    return (torch.zeros((batch_size, num_hiddens), device=device),)

def gru(inputs, state, params):
    # 使用的是有梯度的训练参数
    W_xz, W_hz, b_z, W_xr, W_hr, b_r, W_xh, W_hh, b_h, W_hq, b_q = params
    H, = state
    outputs = []

    # 公式的复现
    # torch.sigmoid将样本值映射到0到1之间。
    for X in inputs:
        Z = torch.sigmoid((X @ W_xz) + (H @ W_hz) + b_z)
        R = torch.sigmoid((X @ W_xr) + (H @ W_hr) + b_r)
        H_tilda = torch.tanh((X @ W_xh) + ((R * H) @ W_hh) + b_h)
        H = Z * H + (1 - Z) * H_tilda
        Y = H @ W_hq + b_q
        outputs.append(Y)
    # torch.cat：将两个张量按指定维度拼接在一起
    return torch.cat(outputs, dim=0), (H,)

vocab_size, num_hiddens, device = len(vocab), 256, d2l.try_gpu()
num_epochs, lr = 500, 1
model = d2l.RNNModelScratch(len(vocab), num_hiddens, device, get_params,
                            init_gru_state, gru)
d2l.train_ch8(model, train_iter, vocab, lr, num_epochs, device)