Recurrent Neural Network (RNN&LSTM)

1. Recurrent Neural Networks

1.1 What is a Recurrent Neural Network?

Recurrent Neural Network (RNN) is a neural network specially used to process sequence data , which can mine timing information and semantic information in data. Utilizing this ability of RNN, the deep learning model has made breakthroughs in solving problems in the field of NLP such as speech recognition, language model, machine translation, and time series analysis.

The flaws of the fully connected neural network in processing sequence data: for example, in the stock forecasting problem, the stock price not only depends on the current input data state, but also depends on historical information, which is impossible for the fully connected neural network.

1.2 Network structure of recurrent neural network

The figure below shows a typical RNN network structure.

In the figure, $U$ is the weight matrix from the input layer to the hidden layer,$W$ is the weight matrix of hidden layer to hidden layer recurrent connections,$V$ is the weight matrix from the hidden layer to the output layer. $x_t$means ttInput at time$t$$h_t$means ttHidden layer vector at time$t$$o_t$means ttoutput at time$t\; .$

$h_t=f(U\cdot x_t+W\cdot h_{t-1}+b_h)$
$o_t=g(V\cdot h_t+b_o)$

where $f(\cdot )$ and$g(\cdot )$ is the activation function.

• Used to calculate the hidden layer vector $h_t$The activation function of is usually chosen in RNNs tanh, and sometimes ReLUis often used;
• Used to calculate the output $o_t$The activation function:
  • If it is a binary classification problem, we may use sigmoidthe function;
  • If it is a k-classification problem, we might use softmaxthe function.

It can be seen that the hidden layer vector of RNN at each moment is not only determined by the input at the current moment, but also depends on the hidden layer vector at the previous moment, which makes RNN have the function of remembering the past time series information.

One thing worth noting is that RNNs share the same weights across different time steps .

1.3 Loss function

In RNN, we define the loss function as the standard logistic regression loss (i.e. cross-entropy loss) .

You can refer to the introduction to cross-entropy in this article .

The output of our neural network ${\hat{y}}^{\left(t\right)}$ is usually some probability value, and$y^{\left(t\right)}$ is a series of determined values ​​(labels), so$y^{(t)}=1$ 。

Loss function for a single word or a single segment in time series data:
$$L^{(t)}({\hat{y}}^{(t)},y^{(t)})=-y^{(t)}log{\hat{y}}^{(t)}=-log{\hat{y}}^{}$$
Overall loss for$${}^{(}$$$${}^t$$$${}^{)}$$
$$L(\hat{y},y)=\frac{1}{T}\sum _{t=1}^T{L}^{(t)}({\hat{y}}^{\left(t\right)},y^{\left(t\right)})$$
and then calculate the parameters in the model by way of back propagation, and use the gradient descent method to update the parameters.

For language models, a good language model can use highly accurate lexical units to predict what we will see next, such as the prompt information of the input method when we type.
In the best case, the model can perfectly estimate the label token with probability 1;
in the worst case, the model predicts the label token with probability 0;
at baseline, the model’s prediction is a uniform distribution over all available tokens in the vocabulary.

1.4 Problems with Recurrent Neural Networks

For length $sequence\; of\; T$ , we compute this TTduring iteration$Gradients\; over\; T$ time steps, which will produce length O ( T ) O(T)during backpropagation$O\left(T\right)$ matrix multiplication chain. When$For\; large\; T$ , this can lead to numerical instability, e.g. possibly exploding or vanishing gradients. In this way we can find that the RNNHas a proximity effect and is not good at capturing long-term dependencies。

2. LSTM

The following content refers to the detailed explanation of LSTM .

2.1 Network structure of LSTM

The full name of LSTM is Long Short Term Memory. The motivation of this algorithm is to solve the long-term dependence problem of RNN mentioned above. The reason why LSTM can solve the long-term dependency problem of RNN is because LSTM introducesgate mechanismUsed to control the flow and loss of features. LSTM is composed of a series of LSTM units (LSTM Unit), and its chain structure is as shown in the figure below.

In the LSTM unit, each yellow box in the figure represents a neural network layer, which consists of weights, biases, and activation functions; each pink circle represents element-level operations; arrows represent vector flow; intersecting arrows represent vector splicing; bifurcated arrows represent vector replication.

2.2 Interpretation of LSTM unit

The core part of LSTM is the part similar to the conveyor belt in the LSTM unit (as shown in the figure below). This part is generally called the cell state (cell state), which exists in the entire chain system of LSTM from beginning to end.

cell state	official
	$C_t=f_t\cdot C_{t-1}+i_t\cdot {\tilde{C}}_t$

in,

• $f_t$is called the forget gate, which means $C_{t-1}$Which features of C t are used to calculate $C_t$ 。$f_t$is a vector, and each element of the vector is in the range [0,1]. Usually we use $sigmo\; idfunction$ as activation function,$The\; output\; of\; the\; sigmoid$ function is a value between [0,1].
• $i_t$is called the input gate and is used to control ${\tilde{C}}_t$Which features of $C_t$. Same as $f_t$It is also a vector with elements between [0,1], usually we use $The\; sigmoid$ function is used as the activation function.
  
  

forget gate $f_t$	official
	$f_t=s(W_f\cdot [h_{t-1},x_t]+b_f)$

input gate $i_t$with cell state update value ${\tilde{C}}_t$	official
	$i_t=s(W_i\cdot [h_{t-1},x_t]+b_i)$ ${\tilde{C}}_t=English(W\_{}_C\cdot [h_{t-1},x_t]+b_C)$

in,

• $i_t$is called the input gate and is used to control ${\tilde{C}}_t$Which features of $C_t$. Same as $f_t$It is also a vector with elements between [0,1], usually we use $The\; sigmoid$ function is used as the activation function.
• ${\tilde{C}}_t$Represents the unit state update value, the activation function usually uses $tyou$ _
  
  

Hidden layer output $h_t$	official
	$o_t=s(W_o\cdot [h_{t-1},x_t]+b_o)$ $h_t=o_t\cdot I\left(older\right){}_t)$

Among them, $o_t$expressoutput gate(Be careful not to confuse with the output in RNN here), the calculation method is the same as $f_t$sum $i_t$same.

---

Reference :
[1] "Deep Learning"
[2] Wu Enda - Deep Learning Course
[3] "Hands-on Deep Learning"
[4] The most detailed explanation of cyclic neural network in history (RNN/LSTM/GRU)
[5] Detailed explanation of LSTM