[Deep Learning] RNN recurrent neural network and LSTM deep learning model

1. Recurrent Neural Network RNN ​​(Recurrent Neural Network)

A core of the recurrent neural network is that it can use the previous sequence data to predict the subsequent results. How to achieve this. The RNN structure is shown in the figure below.

After the information in the front sequence is processed, it is passed to the back sequence as input information.

mathematical model:

$a^1=g_a(W_h{a}^0+W_i{x}^1+b_a)$，$y^1=g_y(W_y{a}^0+W_o{x}^1+b_i)$
$a^2=g_a(W_h{a}^1+W_i{x}^2+b_a)$，$y^2=g_y(W_y{a}^1+W_o{x}^2+b_i)$
……
$a^t=g_a(W_h{a}^{t-1}+W_i{x}^t+b_a)$，$y^t=g_y(W_y{a}^{t-1}+W_o{x}^t+b_i)$

$g$ is the activation function,$W,b$ is the training parameter.

2. Different types of RNN models

Basic RNN model structure

input: $x^1,x^2,x^2,...,x^i$，输出：$y^1,y^2,y^2,...,y^i$
multi-input to multi-output, RNN structure with the same dimensions.
Application: Specific information identification.

Multiple input single output RNN structure

Import: $x^1,x^2,x^2,...,x^i$ , Exit:$y$
Application: Emotion Recognition

Single input multiple output RNN structure

Input: $xi$，输出：$y^1,y^2,y^2,...,y^i$
application: sequence data generator, such as article generation, music generation

Multiple-input multiple-output RNN structure

input: $x^1,x^2,x^2,...,x^i$，输出：$y^1,y^2,y^2,...,y^j$
Application: Language Translation

When the Bidirectional Recurrent Neural Network (BRNN)
makes a judgment, it also takes the posterior sequence information into consideration.

Deep recurrent neural networks (DRNN)
solve more complex sequence tasks, and can be combined with single-layer RNN stacks or full connections before output.

Defects of ordinary RNN structures

First, when the front sequence information is transmitted to the back, the weight of the information decreases, resulting in the loss of important information. The gradient disappears during the solution process.

Second, RNN sometimes loses parameters during the training process.

In the process of minimizing the loss function, the gradient will suddenly and violently jitter, resulting in parameter loss.

So why is this happening?
Take the RNN structure in the figure below as an example, assuming $W^i,W^o$ are all 1, and the input length is 1000, then$y^{1000}=w^{999}$

We assume $The\; initial\; value\; of\; w$ is 1, inwwWhat happens when$w\; changes\; slightly?$

So the problem with RNN is that during the training process, the same$w$ is used repeatedly at different points in time,$Once\; w$ has an impact, it will have a great impact.

Long Short-term Memory Network LSTM (Long Short-term Memory)

Using LSTM can optimize the defects of the RNN structure mentioned above. Add memory cells cic^i to the original ordinary RNN structural unit$c^i$ , can convey information about distant parts of the front.

The LSTM structure consists of three gates, four inputs, and one output. The three gates are input gate, forget gate, and output gate. Use these three gates to control which information should be forgotten and discarded, which information should be retained, or kept unchanged. The four inputs are, input data$Z$ , input gate control signal$Z_i$, forget the gate control signal $Z_f$and output gate control signal $Z_o$。

The door here can be understood as an activation function. This activation function is usually a sigmoid function, because the value of the sigmoid function is between 0 and 1, which is used to control the closing and opening of the door.

Assume that the original value stored in memory $c$ , the updated value is$c^{\prime }=g(z)f(z_i)+cf(z_f)$，$c^{\prime }$ is the new value stored in memory. From this formula we can see that$f(z_i)$ is to control$Can\; Z$ be entered as a level?$f(z_i)=0$ means there is no input,$f\left(z_i\right)=1$ has input. $f\left(z_f\right)$ controls whether the value in memory will be updated. $f\left(z_f\right)=At\; 0$ , forget that the door is open and write 0 into the cell,$f\left(z_f\right)=When\; 1$ , it passes directly, and the cell median value remains unchanged or c. $f\left(z_o\right)$ controls whether there is an output value.

$Z_i,Z_f,Z_{,}{Z}_o$All by XXThe input obtained by multiplying$X\; by\; the\; weight\; matrix.$Enter the sequence as shown below.

LSTM can solve the problem of RNN gradient disappearance. The values ​​in RNN memory will be cleared every time, while the memory in LSTM is always superimposed, and will only be cleared unless the door is closed.