Long short-term memory (LSTM)

introduction

Disadvantages of RNN

RNN was introduced last time, you can refer to the article recurrent neural network (RNN) .
At the end, only the advantages of RNN are mentioned. As a comparison of LSTM, it is necessary to point out its obvious disadvantages. That is, for the representation of historical information, the ht h_t of RNN$h_t$Not too reasonable.
Why? Consider a situation where the activation function of the hidden layer is Relu, or if there is no activation function, then:
$h_t=W(...(W(W{h}_0+U{x}_1)+U{x}_2)...)+U{x}_t$
Then you can expand to get $h_t=W^{t-1}U{x}_1+.......$ . According to this result, we can get:

1. If U is positive, and the weight in W is a number greater than 1, then $h_t$Will be mainly $x_1$Dominant, that is, only x 1 x_1 is saved$x_1$Information.
2. If U is positive and the weight in W is a number from 0 to 1, then $h_t$Will be mainly $x_t$Dominant, $x_1$The information is minimal, $x_2$Second.

In short, according to the above analysis, it seems that RNN will forget sequentially. For example, in the second case above, the farther you choose, the cleaner you will forget.

LSTM improvements

Now, we hope to have a network architecture that can do it, selective forgetting. That is to say, set some more parameters than RNN to do this, in the hope that historical information and current input can be evaluated, if the current input is of no use, a low weight is selectively given to better inherit the history Information; if historical information will not have any influence on the future, then give the historical information a low weight and give the current input a higher weight. How to do it?

LSTM

The network architecture is the same as RNN, but each neuron in the hidden layer is expanded to:

Anatomy

parameter:

Obviously the difference from RNN is that not only the parameters are increased , but also $x_t$And $h_{t-1}$It was reused 4 times.
Which 4 times?


Explanation: the above $f_t,i_t,{The}_t$Yes $[0,$The number of$1$$]$ is a weight.
Used up the above two parameters$x_t$And $h_{t-1}$After that, the next cell memory st s_t needs to be generated$s_t$. This is a significant difference from RNN, this $s_t$And wait for the ht h_t to be calculated$h_t$They must be saved for use by the next neuron.
$s_t$

This means how much $f_t$Forget the memory of the past $s_{t-1}$, To what extent $i_t$Accept the input candidate memory $c_t$。
$h_t$

So far, we found that the historical information is changed by $h_t$And $s_t$The escort has enhanced the model's capabilities.
Finally, output ${with}_t$The formula is:

to sum up

As an innovative improvement based on RNN, LSTM uses three gate mechanisms to achieve selective forgetting, which greatly enhances the expressive power of the network architecture and achieves very good application effects in practice. LSTM has now become a frequently used network, and it is a small expert who deals with time series and contextual problems.