【深度学习】循环神经网络RNN和长短时记忆网络LSTM(超详细原理推导和解释)

写在前面

因为近期要做一些金融股票预测相关的项目课题，最近两天着手看了一下$\mathrm{R}\mathrm{N}\mathrm{N}$和$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$，以前也零零碎碎的看过几次，但都被它庞杂的网络结构和公式吓退了，这次终于静下心来研究了一下这两个网络，并且也亲自用手推导了一下它们的反向传播过程，也辛辛苦苦亲手打了很多公式、画了很多图，花了很长时间，绝对超级详细，只要静下心来，我相信谁都能看懂它们的工作机制尤其是是$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$，最终写下这片文章，一是为更多的人透析其原理提供便利，二是方便自己回过头来复习。里面的内容是自己的理解并结合相关参考资料做的一些笔记，走过路过的朋友们一键三连互关走起来，互相学习，共同促进！再次感谢大佬们的文章和视频！

循环神经网络RNN

引言

Recurrent Neural Network $\left(RecurrentNeuralNetwork\right)referstoastructurethatrecursovertime.\_\_\_\_\_\_\_\_\_\_$ _ It is widely used in many fields such as natural language processing and speech image processing. It is a neural network model for processing sequence data. The traditional neural networkconsists of an input layer, a hidden layer, and an output layer. The output is controlled by an activation function, and the layers are connected by weights. The ultimate goal of neural network training is to learn a set of weights to process new data to meet our requirements. Such as Convolutional Neural Network$\left(CNN\right)$ , the output calculated by forward propagation only considers the influence of the previous input and does not consider the influence of input at other times.

循环神经网络与传统神经网络最大的不同就是，拥有"记忆功能"——可以用来处理基于序列的数据。例如，你要预测一个句子的下一个单词是什么，一般需要用到前面的单词，因为句子中的单词是并不是脱离上下文环境而存在的，我们必须要考虑其所在的上下文环境。也就是说，$\mathrm{R}\mathrm{N}\mathrm{N}$处理的数据当前时刻的输出与前面时刻的输出也有关系。

网络结构

$\mathrm{R}\mathrm{N}\mathrm{N}$的记忆功能具体表现为网络会对前面的信息进行记忆并应用于当前输出的计算中。因此隐藏层之间产生了连接，并且隐藏层的输入不仅包括输入层的输入值，还包括上一时刻隐藏层的输出值。

从上述图中可以看出权重矩阵$W$作用于每一个隐藏层的神经元。其中$x_t$表示当前$t$的输入层输入值，$s_t$表示时刻$t$的隐藏层输出值(下文中用$h_t$表示隐藏层的输出值)，$o_t$则表示时刻 t t Output value of$t$$U$ represents the weight of the input layer,$V$ represents the weight of the output layer.

Weight sharing : It can be seen from the network structure that $U,V,W$ is a set of parameters shared by all features. Its advantage is that in the face of different inputs, different corresponding results can be learned; the number of training parameters is reduced; the length ofinput and output data can be different in different examples.

forward propagation

From the network structure we can see that RNN \rm RNNThe main calculation parameters of$RNN\; are\; the\; output\; value$$of\; the\; hidden\; layer\; and\; the\; value\; of$$the\; output\; layer.$
$$\begin{array}{rl}{h}_t& =f(U{x}_t+W{h}_{t-1})\\ o_t& =g(V{h}_t)\end{array}$$
In the above calculation process, the functions $f,g$ is the activation function,$f$ generally takes$\tanh$ function,$g$ generally takes$softmaxfunction$ .$\_$$\_$$\_$$\_$

通过反复代入
$$\begin{array}{rl}{o}_t& =g(V{h}_t)\\ & =gV(f(U{x}_t+W{h}_{t-1}))\\ & =gV\left(f\right(U{x}_t+Wf(U{x}_{t-1}+W{h}_{t-2})))\\ & =gV\left(f\right(U{x}_t+Wf(U{x}_{t-1}+Wf(U{x}_{t-2}+W{h}_{t-3}))))\\ & =\cdots \end{array}$$
From this it can be seen that ttThe output at time$t$$x_t,x_{t-1},x_{t-2},x_{t-3},...$ are all related.

The bias value bb is omitted here$b$ , generally this value needs to be added.

loss function

A single time step loss can be defined according to the task type, the more widely used is the cross entropy loss $(\mathrm{C}\mathrm{E})$，即
$$L_{\mathrm{C}\mathrm{E}}=-(y_t\ln o_t+(1-y_t)\ln (1-o_t))$$
整个时间序列的损失就是单个时间步的损失之和，即
$$L=\sum _t{L}_{\mathrm{C}\mathrm{E}}$$
其中，$y_t$表示时刻$t$的真实标签值，$o_t$表示时刻$t$模型预测输出值。

反向传播

从前向传播的过程中，可以看出只需要对三个权值$U,V,W$进行优化即可，因此分别对其求梯度。

参数优化

• 对参数$V$求偏导
  $$\frac{\partial L}{\partial V}=\sum _t\frac{\partial L_t}{\partial o_t}\frac{\partial o_t}{\partial V}$$
  其中还要对复合函数(激活函数)求导。

• 对参数$W$求偏导

  对该参数涉及到之前时刻的信息，求导相对比较复杂，因此假设在$t=3$时刻，利用前面时刻的数据对$W$求导，即

  $$\begin{array}{rl}\frac{\partial L_3}{\partial W}& =\frac{\partial L_3}{\partial o_3}\frac{\partial o_3}{\partial h_3}(\frac{\partial h_3}{\partial W}+\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial W}+\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial h_1}\frac{\partial h_1}{\partial W})\\ & =\frac{\partial L_t}{\partial o_t}\frac{\partial o_t}{\partial h_t}\sum _{j=1}^t\left[\left(\prod _{i=j+1}^t\frac{\partial h_i}{\partial h_{i-1}}\right)\frac{\partial h_j}{\partial W}\right]\end{array}$$

• For parameter $U$ for partial derivative

  Assuming ditto. Therefore there

  $$\begin{array}{rl}\frac{\partial L_3}{\partial W}& =\frac{\partial L_3}{\partial o_3}\frac{\partial o_3}{\partial h_3}(\frac{\partial h_3}{\partial U}+\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial U}+\frac{\partial h_3}{\partial h_2}\frac{\partial h_2}{\partial h_1}\frac{\partial h_1}{\partial U})\\ & =\frac{\partial L_t}{\partial o_t}\frac{\partial o_t}{\partial h_t}\sum _{j=1}^t\left[\left(\prod _{i=j+1}^t\frac{\partial h_i}{\partial h_{i-1}}\right)\frac{\partial h_j}{\partial U}\right]\end{array}$$

梯度消失或爆炸

在参数优化过程中，随着时间的不断累积，对参数$U,W$的优化过程就会出现梯度累乘，此间要对函数$f$求导，因此有
$$\prod _{i=j+1}^t\frac{\partial h_i}{\partial h_{i-1}}=\prod _{i=j+1}^t{f}^{\prime }\cdot W$$
就会导致激活函数的累乘，由于激活函数通常为$\tanh$或者$\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}$，其函数和导数图像为

由此可知，不管使用哪种激活函数，其导数的值总不会超过$1$，累乘之后就会出现梯度消失的问题。因此，只有两种解决办法，其一就是使用更好的激活函数(比如$\mathrm{R}\mathrm{e}\mathrm{L}\mathrm{U}$)，其二就是改变网络的传播结构，也就是后来提出的$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$。

长短时记忆网络LSTM

网络结构以及与RNN的区别

长短时记忆网络$(\mathrm{L}\mathrm{o}\mathrm{n}\mathrm{g}\mathrm{s}\mathrm{h}\mathrm{o}\mathrm{r}\mathrm{t}-\mathrm{t}\mathrm{e}\mathrm{r}\mathrm{m}\mathrm{m}\mathrm{e}\mathrm{m}\mathrm{o}\mathrm{r}\mathrm{y})$，是一种特殊的循环神经网络，它通过特殊的传播结构设计来避免长期依赖，从而缓解$\mathrm{R}\mathrm{N}\mathrm{N}$梯度消失或者爆炸的情形，它在更长的序列中能有更好的表现。$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$和循环神经网络有基本相同的网络结构，唯一不同的就是单层的前向传播网络模块，单一重复模块有$4$个网络层，以一种特殊的方式进行交互。本质上的不同就是$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$通过记忆细胞选择性记忆重要信息，过滤掉不重要的信息，减轻记忆负担，而$\mathrm{R}\mathrm{N}\mathrm{N}$则记住所有信息，增加了网络的负担。

核心思想

$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$的关键就是细胞状态，也就是下图中的$C_t$，细胞状态在相当于一个传送带上传送，只有少量的线性交互，所以信息很难流传或者长时间记忆而不发生改变。它主要就是通过称为"门"的结构来控制信号传播以调节细胞状态，从而实现在细胞状态中添加或者删减信息，其中，"门"就是上述结构中的一些相乘或者相加的结构，通过激活函数来使信息选择性通过。

前向传播

遗忘门

上图所示的是$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$的第一阶段，也就是"忘记阶段"，具体来说，就是通过$h_{t-1}$和$x_t$计算出$f_t(\mathrm{f}\mathrm{o}\mathrm{r}\mathrm{g}\mathrm{e}\mathrm{t})$作为忘记门控，忘记来自细胞状态$C_{t-1}$中不重要的信息，即对于来此$C_{t-1}$状态的每个数输出$0$到$1$之间的数，$1$表示完全记住，$0$表示完全忘记。

语言模型的例子中，基于已经看到的预测下一个词。在这个问题中，细胞状态可能包含当前主语的性别，因此正确的代词可以被选择出来。当看到新的主语，希望忘记旧的主语。

$f_t$具体计算方式如下
$$f_t=\sigma (W_{xf}{x}_t+W_{hf}{h}_{t-1}+b_f)$$

输入门

上图所示的是$\mathrm{L}\mathrm{S}\mathrm{T}\mathrm{M}$第二阶段，也就是"输入阶段"，就是决定在细胞状态中增加什么信息。其中有两部分，第一部分就是通过$sigmoid$ function todetermine what new information to add, the second part is through$\tanh$ function to create a candidate value vector$\widetilde{C_t}$。

具体计算方式如下
$$\begin{array}{rl}{i}_t& =s(W_{xi}{x}_t+W_{hi}{h}_{t-1}+b_i)\\ \widetilde{C_t}& =\mathrm{fishy}(W_{xC}{x}_t+W_{hC}{h}_{t-1}+b_C)\end{array}$$

状态更新

将旧的细胞状态$C_{t-1}$更新为$C_t$，前面的阶段我们已经确定了要遗忘、记住和添加的信息，现在就是实际去完成这个操作。

具体计算方式如下
$$C_t=f_t\odot C_{t-1}+i_t\odot \widetilde{C_t}$$
用$f_t$乘以旧的状态$C_{t-1}$来忘记决定忘记的信息，再加上$i_t\odot \widetilde{C_t}$，它是新的候选值，根据我们决定更新每个状态的程度进行变化。

输出门

如上图所示，该阶段会决定输出什么值，输出会基于当前细胞状态$C_t$，也是一个过滤之后的版本。首先通过$\mathrm{s}\mathrm{i}\mathrm{g}\mathrm{m}\mathrm{o}\mathrm{i}\mathrm{d}$函数决定当前状态下的哪些输入需要输出，然后将当前细胞状态$C_t$通过$\tanh$函数压缩到$-1$到$1$之间，并将它和$o_t$进行相乘，最终输出我们分。

具体计算方式如下
$$\begin{array}{rl}{o}_t& =\sigma (W_{xo}{x}_t+W_{ho}{h}_{t-1}+b_o)\\ h_t& =o_t\odot \tanh (C_t)\end{array}$$

反向传播

首先将前向传播的表达式罗列如下：
$$\{\begin{array}{ll}{f}_t& =s(W_{xf}{x}_t+W_{hf}{h}_{t-1}+b_f)\\ i_t& =s(W_{xi}{x}_t+W_{hi}{h}_{t-1}+b_i)\\ {\tilde{C}}_t& =\mathrm{fishy}(W_{x\tilde{C}}{x}_t+W_{h\tilde{C}}{h}_{t-1}+b_{\tilde{C}})\\ C_t& =f_t\odot C_{t-1}+i_t\odot {\tilde{C}}_t\\ o_t& =s(W_{xo}{x}_t+W_h{h}_{t-1}+b_o)\\ h_t& =o_t\odot \mathrm{fishy}\left({\_}_t\right)\\ y_t& =W_y{h}_t+b_y\end{array}$$

Parameter optimization

It can be seen from the process of forward propagation that the parameters that need to be optimized are only $W_{xf},W_{hf},W_{xi},W_{hi},W_{xC},W_{hC}{W}_{xo},W_h,Wy$，因此基于它们求导。参照$\mathrm{R}\mathrm{N}\mathrm{N}$的反向传播，假设在时刻$t=3$，利用之前时刻的数据对上述优化参数中的$W_{xf}$求
$$\frac{\partial L_3}{\partial W_{xf}^{(3)}}+\frac{\partial L_3}{\partial W_{xf}^{(2)}}+\frac{\partial L_3}{\partial W_{xf}^{(1)}}$$

$$\frac{\partial L_3}{\partial W_{xf}^{(3)}}=\frac{\partial L_3}{\partial y_3}\frac{\partial y_3}{\partial h_3}\frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial f_3}\frac{\partial f_3}{\partial W_{xf}^{\left(3\right)}}$$
$$\begin{array}{rl}\frac{\partial L_3}{\partial W_{xf}^{(2)}}& =\frac{\partial L_3}{\partial y_3}\frac{\partial y_3}{\partial h_3}\left\{\begin{array}{l}\frac{\partial h_3}{\partial o_3}\frac{\partial o_3}{\partial h_2}\frac{\partial h_2}{\partial C_2}\\ \frac{\partial h_3}{\partial C_3}\left\{\begin{array}{l}\frac{\partial C_3}{\partial C_2}\\ \frac{\partial C_3}{\partial f_3}\frac{\partial f_3}{\partial h_2}\frac{\partial h_2}{\partial C_2}\\ \frac{\partial C_3}{\partial i_3}\frac{\partial i_3}{\partial h_2}\frac{\partial h_2}{\partial C_2}\\ \frac{\partial C_3}{\partial {\tilde{C}}_3}\frac{\partial {\tilde{C}}_3}{\partial h_2}\frac{\partial h_2}{\partial C_2}\end{array}\right\}\end{array}\right\}\frac{\partial C_2}{\partial f_2}\frac{\partial f_2}{\partial W_{xf}^{\left(2\right)}}\\ & =\frac{\partial L_3}{\partial y_3}\frac{\partial y_3}{\partial h_3}\left\{\begin{array}{l}\frac{\partial h_3}{\partial o_3}\frac{\partial o_3}{\partial h_2}\frac{\partial h_2}{\partial C_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial C_2}\end{array}\right\}\frac{\partial C_2}{\partial f_2}\frac{\partial f_2}{\partial W_{xf}^{\left(2\right)}}\end{array}$$

$$t_3\to t_1=\left\{\begin{array}{l}{t}_3\to t_2:h_3\to \left\{\begin{array}{l}{o}_3\to h_2\\ C_3\to C_2\\ C_3\to f_3\to h_2\\ C_3\to i_3\to h_2\\ C_3\to {\tilde{C}}_3\to h_2\end{array}\right\}\\ t_2\to t_1\left\{\begin{array}{l}{C}_2\to \left\{\begin{array}{l}{f}_2\to h_1\to C_1\to f_1\\ i_2\to h_1\to C_1\to f_1\\ {\tilde{C}}_2\to h_1\to C_1\to f_1\\ C_1\to f_1\end{array}\right\}\\ h_2\to \left\{\begin{array}{l}{o}_2\to h_1\to C_1\to f_1\\ C_2\to C_1\to f_1\\ C_2\to f_2\to h_1\to C_1\to f_1\\ C_2\to i_2\to h_1\to C_1\to f_1\\ C_2\to {\tilde{C}}_2\to h_1\to C_1\to f_1\end{array}\right\}\end{array}\right\}\end{array}\right\}{t}_3\to t_2\to t_1:\mathrm{t}\mathrm{o}\mathrm{t}\mathrm{a}\mathrm{l}24\mathrm{p}\mathrm{a}\mathrm{t}\mathrm{h}\mathrm{s}$$

$$\begin{array}{rl}\frac{\partial L_3}{\partial W_{xf}^{(1)}}& =\frac{\partial L_3}{\partial y_3}\frac{\partial y_3}{\partial h_3}\left\{\begin{array}{l}{t}_3\to t_2\left\{\begin{array}{l}\frac{\partial h_3}{\partial o_3}\frac{\partial o_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial C_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial f_3}\frac{\partial f_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial i_3}\frac{\partial i_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial {\tilde{C}}_3}\frac{\partial {\tilde{C}}_3}{\partial h_2}\end{array}\right\}\\ t_2\to t_1\left\{\begin{array}{l}\left\{\begin{array}{l}\frac{\partial C_2}{\partial f_2}\frac{\partial f_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\\ \frac{\partial C_2}{\partial i_2}\frac{\partial i_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\\ \frac{\partial C_2}{\partial {\tilde{C}}_2}\frac{\partial {\tilde{C}}_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\\ \frac{\partial C_2}{\partial C_1}\end{array}\right\}\\ \left\{\begin{array}{l}\frac{\partial h_2}{\partial o_2}\frac{\partial o_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\\ \frac{\partial h_2}{\partial C_2}\left\{\begin{array}{l}\frac{\partial C_2}{\partial C_1}\\ \frac{\partial C_2}{\partial f_2}\frac{\partial f_2}{\partial C_1}\\ \frac{\partial C_2}{\partial i_2}\frac{\partial i_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\\ \frac{\partial C_2}{\partial {\tilde{C}}_2}\frac{\partial {\tilde{C}}_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\end{array}\right\}\end{array}\right\}\end{array}\right\}\frac{\partial C_1}{\partial f_1}\end{array}\right\}\frac{\partial f_1}{\partial W_{xf}^{\left(1\right)}}\\ & =\frac{\partial L_3}{\partial y_3}\frac{\partial y_3}{\partial h_3}\left\{\begin{array}{l}{t}_3\to t_2\left\{\begin{array}{l}\frac{\partial h_3}{\partial o_3}\frac{\partial o_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial C_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial f_3}\frac{\partial f_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial i_3}\frac{\partial i_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial {\tilde{C}}_3}\frac{\partial {\tilde{C}}_3}{\partial h_2}\end{array}\right\}\\ t_2\to t_1\left\{\begin{array}{l}\frac{\partial C_2}{\partial C_1}\\ \left\{\begin{array}{l}\frac{\partial h_2}{\partial o_2}\frac{\partial o_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\\ \frac{\partial h_2}{\partial C_2}\frac{\partial C_2}{\partial C_1}\end{array}\right\}\end{array}\right\}\frac{\partial C_1}{\partial f_1}\end{array}\right\}\frac{\partial f_1}{\partial W_{xf}^{\left(1\right)}}\\ & =\frac{\partial L_3}{\partial y_3}\frac{\partial y_3}{\partial h_3}\left\{\begin{array}{l}\frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial C_2}\frac{\partial C_2}{\partial C_1}\\ \left\{\begin{array}{l}\frac{\partial h_3}{\partial o_3}\frac{\partial o_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial f_3}\frac{\partial f_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial i_3}\frac{\partial i_3}{\partial h_2}\\ \frac{\partial h_3}{\partial C_3}\frac{\partial C_3}{\partial {\tilde{C}}_3}\frac{\partial {\tilde{C}}_3}{\partial h_2}\end{array}\right\}\frac{\partial h_2}{\partial o_2}\frac{\partial o_2}{\partial h_1}\frac{\partial h_1}{\partial C_1}\\ \frac{\partial h_3}{\partial o_3}\frac{\partial o_3}{\partial h_2}\frac{\partial h_2}{\partial C_2}\frac{\partial C_2}{\partial C_1}\end{array}\right\}\frac{\partial C_1}{\partial f_1}\frac{\partial f_1}{\partial W_{xf}^{\left(1\right)}}\end{array}$$

From the above calculation process, we can see that in the final derivation result, there will be a large number of cumulative multiplications in the following form,
$$\cdots \prod _{t=m}^n\frac{\partial C_t}{\partial C_{t-1}}\cdots$$

in,
$$\frac{\partial C_t}{\partial C_{t-1}}=\oplus \{\begin{array}{l}\frac{\partial C_t}{\partial f_t}\frac{\partial f_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}}\left(=C_{t-1}\cdot W_{hf}{p}^{\prime }\cdot o_{t-1}{\mathrm{fishy}}^{\prime }\right)\\ \frac{\partial C_t}{\partial i_t}\frac{\partial i_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}}\left(={\tilde{C}}_t\cdot W_{hi}{p}^{\prime }\cdot o_{t-1}{\mathrm{fishy}}^{\prime }\right)\\ \frac{\partial C_t}{\partial {\tilde{C}}_t}\frac{\partial {\tilde{C}}_t}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial C_{t-1}}\left(=i_t\cdot W_{h\tilde{C}}{\mathrm{fishy}}^{\prime }\cdot o_{t-1}\cdot {\mathrm{fishy}}^{\prime }\right)\end{array}$$

It can be seen from the above formula that $\frac{\partial C_t}{\partial C_{t-1}}$The value of can be adjusted by parameters $W_{hf},W_{hi},W_{h\tilde{C}}$To control flexibly, in order to prevent the gradient from disappearing, it can be controlled at $1$ near, then there will be1 1In the case of multiplication by$1$$t=1,2,3.$ When more moments are considered, more and more multiplications will appear. In addition to being controlled by theabove-mentioned multiplications, other forms of gradient multiplications may make the gradient disappear, so$\frac{\partial C_t}{\partial C_{t-1}}$The cumulative multiplication greatly alleviates the problem of gradient disappearance.

from $The\; working\; mechanism\; of\; LSTM$ is explained, that is,$t=m$ to$t=$In the short period oftime$n$ , the information remembered in the cell state is basically the same, which will also make the gradients of two adjacent moments almost the same, thereby relieving the gradient from disappearing.

References

[1] Video at Station B: [Revisiting the classics] Explaining how the LSTM long-term and short-term memory network alleviates the disappearance of gradients in vernacular, and deriving backpropagation with hands-on formulas

[2] Introduction to LSTM and derivation of backpropagation algorithm that everyone can understand (very detailed)

[3] Understanding LSTM Networks

[4] Understanding LSTM Networks