Chapter 6 Recurrent Neural Networks

Series Article Directory

Chapter 1 Introduction
Chapter 2 Overview of Machine Learning
Chapter 3 Linear Model
Chapter 4 Feedforward Neural Network
Chapter 5 Convolutional Neural Network Chapter 6
Recurrent Neural Network
Chapter 7 Network Optimization and Regularization
Chapter 8 Attention Mechanism and External Memory
Chapter 9 Unsupervised Learning
Chapter 10 Model Independent Learning
Chapter 11 Probabilistic Graphical Models
Chapter 12 Deep Belief Networks
Chapter 13 Deep Generative Models
Chapter 14 Deep Reinforcement Learning
Chapter 15 Sequence Generative Models

---

---

foreword

This article introduces recurrent neural networks.

---

6.1 Adding memory to neural networks

6.1.1 Delay Neural Network

Time Delay Neural Network (TDNN), that is, to establish an additional delay unit to store the historical information of the network (including input, output, hidden state, etc.).
$$h_t^{(l)}=f(h_t^{(l-1)},h_{t-1}^{(l-1)},\dots h_{t-K}^{(l-1)})$$

6.1.2 Autoregressive models

Autoregressive Model (AR), a type of time series model, uses historical information of variables to predict itself.
$$y_t=w_0+\sum _{k=1}^K{w}_k{y}_{t-k}+{ϵ}_t$$

${ϵ}_t$is the noise at the tth moment

6.1.3 Nonlinear autoregressive models

Nonlinear Autoregressive with Exogenous Inputs Model (NARX)
$$y_t=f(x_t,x_{t-1},\dots ，x_{t-K_x},y_{t-1},y_{t-2},\dots ，y_{t-K_x})$$
where f(⋅) represents a nonlinear function, which can be a feed-forward network,$K_x$和$K_y$as a hyperparameter

6.2 Recurrent Neural Networks

6.2.1 Network structure

$$h_t=f(h_{t-1},x_t)$$

• Recurrent neural networks are capable of processing time-series data of arbitrary length by using neurons with self-feedback.

• Recurrent neural networks are more consistent with the structure of biological neural networks than feedforward neural networks.

• Recurrent neural networks have been widely used in tasks such as speech recognition, language models, and natural language generation.

6.2.2 Network Expansion in Time

6.2.3 Simple Recurrent Networks

State update function :
$$h_t=f(U{h}_{t-1}+W{x}_t+b)$$
General Approximation Theorem:

6.2.4 Turing Complete

Turing Completeness refers to a data manipulation rule, such as a computer programming language, which can realize all the functions of a Turing machine and solve all computable problems.

A fully connected recurrent neural network can approximately solve all computable problems.

6.2.5 Applications

• A machine learning model as an input-output mapping (this section focuses on this case).
• As an associative memory model in memory.

6.3 Applications of Recurrent Neural Networks in Machine Learning

6.3.1 Sequence to Category

Model Architecture :

Practical Applications :
Sentiment Classification Tasks

6.3.2 Synchronized Sequence-to-Sequence Mode

Model Structure :

Practical Application :

1. Chinese participle

2. Information Extraction (IE)

Extract structured information from unstructured text to form knowledge.

3. Speech recognition

6.3.3 Asynchronous sequence-to-sequence mode

Model Structure :

Application :
Machine Translation

${ϵ}_t$is the noise at the tth moment

6.4 Gradients

6.4.1 Definition

Given a training sample (x,y), where x = ( $x_1$,… ,$x_T$) is the input sequence whose length is T, y = ( $y_1$ ,… ,$y_T$) is a tag sequence of length T. Then the instantaneous loss function at time t is:
$$L_t=L(y_t,g(h_t\left)\right)$$
The total loss function is:
$$L=\sum _{t=1}^T{L}_t$$

6.4.2 Calculation

Backpropagation through time algorithm:

Vanishing and Exploding Gradients:

Due to the calculation process $The\; \lambda$ part needs to be quadratured many times, so when the obtained value is greater than 1, the problem of gradient explosion will occur, and when it is less than 1, the problem of gradient disappearance will occur.

6.4.3 Long-range dependency problem

The reason for the occurrence :
the cyclic neural network is very deep in the time dimension, so there will be gradient disappearance or explosion problems, so in fact only short-term dependencies can be learned. This is the so-called long-range dependency problem .

Improvement principle :
For the gradient explosion problem, the weight decay or gradient truncation method can be used; for the gradient disappearance problem, the model can be improved.
ways to improve

1. The loop edge is changed to a linear dependency:
   $$h_t=h_{t-1}+g(x_t;i)$$
2. Add nonlinearity:
   $$h_t=h_{t-1}+g(x_t;h_{t-1};i)$$
3. Use the door controller:
   control the accumulation speed of information, including selectively adding new information and selectively forgetting previously accumulated information.
   $$h_t=h_{t-1}+g(x_t;h_{t-1};i)$$

6.5 GRU and LSTM

6.5.1GRU（Gated Recurrent Unit）

1. structure diagram
   
2. 计算公式
   $$\begin{gathered} r_t=s(W_r{x}_t+U_r{h}_{t-1}+b_r) \\ {z}_t=s(W_r{x}_t+U_z{h}_{t-1}+b_z) \\ {\tilde{h}}_t=tanh(W_c{x}_t+U(r_t\odot h_{t-1})) \\ {h}_t=z_t\odot h_{t-1}+(1-z_t)\odot {\tilde{h}}_t \end{gathered}$$

6.5.2 LSTM(Long Short-Term Memory)

1. structure diagram
   

2. 计算公式
   $$\begin{gathered} f_t=s(W_f{x}_t+U_f{h}_{t-1}+b_f) \\ {i}_t=s(W_i{x}_t+U_i{h}_{t-1}+b_i) \\ {\tilde{c}}_t=tanh(W_c{x}_t+U_c{h}_{t-1}+b_c) \\ {o}_t=s(W_o{x}_t+U_o{h}_{t-1}+b_o) \\ {c}_t=f_t\odot c_{t-1}+i_t\odot {\tilde{c}}_t \\ {h}_t=o_t\odot tanh\left(c_t\right) \end{gathered}$$

3. Variations of LSTMs
   

6.6 Deep Models

Stacked Recurrent Neural Networks :

Bidirectional Recurrent Neural Networks :

6.7 Graph Networks

6.7.1 Recurrent Neural Networks

Recurrent neural networks actually share a composition function on a directed graph acyclic graph:

Degenerates to a recurrent neural network:

Natural language processing:

6.7.2 Graph Networks

Calculation process:

Function Compute:

6.7 Applications of Recurrent Networks

Judging the rationality of sentences, composing lyrics, composing poems, machine translation, speaking by looking at pictures, writing, and dialogue systems.

---

Summarize