Introduction to Deep Learning (59) Recurrent Neural Networks - Backpropagation through Time

foreword

The core content comes from blog link 1 blog link 2 I hope you can support the author a lot
This article is used for records to prevent forgetting

Recurrent Neural Networks - Backpropagation Through Time

Textbook

So far, we've repeatedly mentioned things like 梯度爆炸or 梯度消失, and the need for recurrent neural networks 分离梯度. For example, in the RNN implementation from scratch section, we called detachthe function on the sequence. In order to be able to quickly build a model and understand how it works, none of the concepts mentioned above have been fully explained. This section explores the details of backpropagation for sequence models in more depth, as well as the associated mathematics.

When we first implemented recurrent neural networks, we encountered 梯度爆炸problems. We found that gradient truncation is critical to ensure model convergence. In order to better understand this problem, this section will review how the gradient of the sequence model is calculated. There is nothing new in its working principle. After all, we are still using the chain rule to calculate the gradient.

We previously described forward and backpropagation and the associated computational graphs in a multilayer perceptron. Forward propagation in recurrent neural networks is relatively simple. Passing 时间反向传播（backpropagation through time，BPTT）is actually a specific application of the backpropagation technique in recurrent neural networks. It requires us to unroll the computational graph of a recurrent neural network one time step at a time to obtain dependencies between model variables and parameters. Then, based on the chain rule, backpropagation is applied to compute and store gradients. As sequences can be quite long, dependencies can also be quite long. For example, in a sequence of 1000 characters, the first token may have a significant effect on the last token. This is computationally infeasible (it requires too much time and memory), and requires over 1000 matrix products to get the very elusive gradient. This process is full of computational and statistical uncertainties. In the following, we clarify what happens and how to resolve them in practice.

1 Gradient Analysis of Recurrent Neural Network

We start with a simplified model of how a recurrent neural network works, which ignores the details of the properties of the hidden state and how it is updated. The mathematical representation here does not distinguish scalars, vectors, and matrices as clearly as in the past, because these details are not important for the analysis and instead only confuse the notation in this subsection.

In this simplified model, we set the time step $The\; hidden\; state\; of\; t$ is expressed as$h_t$, the input is expressed as $x_t$, the output is expressed as $o_t$. Recall from our discussion in the RNN section that the input and hidden state can be concatenated and multiplied by a weight variable in the hidden layer. Therefore, we use wh w_h respectively$w_h$和$w_o$to represent the weights of the hidden layer and the output layer. The hidden state and output of each time step can be written as:
$$\begin{array}{rl}{h}_t& =f(x_t,h_{t-1},w_h),\\ o_t& =g(h_t,w_o),\end{array}$$

where $f$ and$g$ are the transformations of the hidden layer and the output layer, respectively. Thus, we have a chain{ … , ( xt − 1 , ht − 1 , ot − 1 ) , ( xt , ht , ot ) , … } \{\ldots, (x_{t-1}, h_{t- 1}, o_{t-1}), (x_{t}, h_{t}, o_t), \ldots\}{ …,(xt−1​,ht−1​,ot−1​),(xt​,ht​,ot​),… } , which depend on each other through loop calculations. Forward propagation is fairly simple, traversing the triplet$(x_t,h_t,o_t)$ , and then pass an objective function over allTTEvaluate the output ot o_t$over\; T$ time steps$o_t$and the corresponding label $y_t$之间的差异：
$$L(x_1,\dots ,x_T,y_1,\dots ,y_T,w_h,w_o)=\frac{1}{T}\sum _{t=1}^{T}l(y_t,o_t).$$
For backpropagation, the problem is a bit trickier, especially when we compute the objective function$L$ about the parameter$w_h$的梯度时。 具体来说，按照链式法则：
$$\begin{array}{r}\begin{array}{rl}\frac{\partial L}{\partial w_h}& =\frac{1}{T}\sum _{t=1}^T\frac{\partial l(y_t,o_t)}{\partial w_h}\\ & =\frac{1}{T}\sum _{t=1}^T\frac{\partial l(y_t,o_t)}{\partial o_t}\frac{\partial g(h_t,w_o)}{\partial h_t}\frac{\partial h_t}{\partial w_h}.\end{array}\end{array}$$
The first and second terms of the product in the above formula are easy to calculate, while the third term $\partial h_t/\partial w_h$is where things get tricky, because we need to recursively compute the parameter $w_h$to $h_t$的影响。 根据 $\begin{array}{rl}{h}_t& =f(x_t,h_{t-1},w_h),\\ o_t& =g(h_t,w_o),\end{array}$Recursive computation in $h_t$Both depends on $h_{t-1}$which in turn depends on $w_h$, where $h_{t-1}$The calculation of also depends on $w_h$。 因此，使用链式法则产生：
$$\frac{\partial h_t}{\partial w_h}=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+\frac{\partial f(x_t,h_{t-1},w_h)}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial w_h}.In$$
order to derive the above gradient, suppose we have three sequences{ at } , { bt } , { ct } \{a_{t}\},\{b_{t}\},\{c_{t}\}{ at​},{ bt​},{ ct​}， 当$t=1,2,...$ , the sequence satisfies$a_0=0$且$a_t=b_t+c_t{a}_{t-1}$. for $t\ge 1$，就很容易得出：
$$a_t=b_t+\sum _{i=1}^{t-1}\left(\prod _{j=i+1}^t{c}_j\right)b_i.replace$$ at a_t
based on the following formula$a_t$、$b_t$and $c_t$：
$$\begin{array}{r}\begin{array}{rl}{a}_t& =\frac{\partial h_t}{\partial w_h},\\ b_t& =\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h},\\ c_t& =\frac{\partial f(x_t,h_{t-1},w_h)}{\partial h_{t-1}},\end{array}\end{array}$$

公式$\frac{\partial h_t}{\partial w_h}=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+\frac{\partial f(x_t,h_{t-1},w_h)}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial w_h}$The gradient calculation in$.$$a_t=b_t+c_t{a}_{t-1}$。 因此，对于每个 $a_t=b_t+{\sum }_{i=1}^{t-1}\left({\prod }_{j=i+1}^t{c}_j\right)b_i.$ , we can remove$\frac{\partial h_t}{\partial w_h}=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+\frac{\partial f(x_t,h_{t-1},w_h)}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial w_h}.$中的循环计算
$$\frac{\partial h_t}{\partial w_h}=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+\sum _{i=1}^{t-1}\left(\prod _{j=i+1}^t\frac{\partial f(x_j,h_{j-1},w_h)}{\partial h_{j-1}}\right)\frac{\partial f(x_i,h_{i-1},w_h)}{\partial w_h}.While$$
we can recursively compute$\partial h_t/\partial w_h$, but the chain becomes very long when $t is large. We need to find a way to deal with this problem.

1.1 Complete Computation

显然，我们可以仅仅计算$\frac{\partial h_t}{\partial w_h}=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+{\sum }_{i=1}^{t-1}\left({\prod }_{j=i+1}^t\frac{\partial f(x_j,h_{j-1},w_h)}{\partial h_{j-1}}\right)\frac{\partial f(x_i,h_{i-1},w_h)}{\partial w_h}.$ However, such computations are very slow and gradient explosions may occur, since small changes in the initial conditions may have a large impact on the results. That is, we can observe a phenomenon similar to the butterfly effect, where small changes in initial conditions lead to disproportionate changes in outcomes. This is highly undesirable for the model we want to estimate. After all, what we are looking for is an estimator that generalizes well to models with high stability. Therefore, in practice, this method is almost never used.

1.2 Truncation time step

Alternatively, we can use $\tau$步后截断$\frac{\partial h_t}{\partial w_h}=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+{\sum }_{i=1}^{t-1}\left({\prod }_{j=i+1}^t\frac{\partial f(x_j,h_{j-1},w_h)}{\partial h_{j-1}}\right)\frac{\partial f(x_i,h_{i-1},w_h)}{\partial w_h}$The sum calculation in$.$This is what we've been discussing so far, e.g. when separating gradients in the RNN implementation from zero section. This leads to the true gradient近似, just terminate the summation as$\partial h_{t-}/\partial w_h$. In practice, this approach works pretty well. It is often called truncated backpropagation through time. Doing so resulted in the model focusing primarily on short-term effects rather than long-term effects. This is desirable in reality because it biases the estimates towards simpler and more stable models

1.3 Random truncation

Finally, we can replace ∂ ht / ∂ wh \partial h_t/\partial w_h with a random variable$\partial h_t/\partial w_h$, the random variable is correct as expected, but truncates the sequence. This random variable is obtained by using the sequence $X_t$To achieve, the sequence is predefined $0\le {Pi}_t\le 1$ , where$P(x_t=0)=1-{Pi}_t$且$P(x_t={Pi}_t^{-1})={Pi}_t$, so $E[x_t]=1$。 我们使用它来替换$\frac{\partial h_t}{\partial w_h}=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+\frac{\partial f(x_t,h_{t-1},w_h)}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial w_h}The\; gradient$ in$\partial h_t/\partial w_h$得到：
$$z_t=\frac{\partial f(x_t,h_{t-1},w_h)}{\partial w_h}+X_t\frac{\partial f(x_t,h_{t-1},w_h)}{\partial h_{t-1}}\frac{\partial h_{t-1}}{\partial w_h}.from$$ ξ t \
xi_t$X_t$Derived from the definition of $E[z_t]=\partial h_t/\partial w_h$. Whenever $X_t=0$ , the recursive calculation terminates at this$t$ time step. This results in a weighted sum of sequences of different lengths, where long sequences occur infrequently and so will be weighted appropriately.

1.4 Comparison Strategies

Comparing the strategy of calculating the gradient in RNN, the 3 lines from top to bottom are: random truncation, regular truncation, and complete calculation

The figure above illustrates three strategies for analyzing the first few characters in The Time Machine when using backpropagation through time based on recurrent neural networks:

• The first line is randomly truncated by dividing the text into fragments of different lengths;

• The second line applies regular truncation by breaking the text into subsequences of equal length. This is what we have been doing in our recurrent neural network experiments;

• The third line employs full backpropagation through time, resulting in expressions that are computationally infeasible.

Unfortunately, while random truncation is attractive in theory, it is probably no better in practice than regular truncation due to a variety of factors. First, the observations, after backpropagating several time steps in the past, are sufficient to capture the actual dependencies. Second, the increased variance offsets the fact that gradients are more accurate with more time steps. Third, what we really want are models with only short-range interactions. Therefore, what the model needs is the mild regularization effect of the truncated backpropagation through time method.

2 Details of backpropagation through time

After discussing the general principles, we look at the details of the backpropagation through time problem. Unlike the analysis above, below we show how to compute the gradients of the objective function with respect to all decomposition model parameters. To keep it simple, we consider a recurrent neural network with no bias parameters whose activation function in the hidden layer uses the identity map $\varphi \left(x\right)=x$ . For time step$t$ , let the input of a single sample and its corresponding label be${\mathbf{x}}_t\in {\mathbb{R}}^d$和$y_t$. Calculate the hidden state ${\mathbf{h}}_t\in R^h$ and output${\mathbf{o}}_t\in R^{qState}$ :
$$\begin{array}{rl}{h}_t& ={\mathbf{W}}_{hx}{x}_t+W_{hh}{h}_{t-1},\\ o_t& =W_{qh}{h}_t,\end{array}$$
Where the weight parameter is $W_{hx}\in R^{h\times d}$、${\mathbf{W}}_{hh}\in {\mathbb{R}}^{h\times h}$ 和$W_{qh}\in R^{q\times h}$。 用$l({\mathbf{o}}_t,y_t)$ represents the time step$at\; t$ (i.e. beyond TTfrom the beginning of the sequence$T$ time steps), then the overall loss of our objective function is:
$$L=\frac{1}{T}\sum _{t=1}^{T}l(o_t,y_t).$$

In order to visualize the dependencies between model variables and parameters during the calculation of the recurrent neural network, we can draw a calculation graph for the model, as shown in the figure below. For example, the hidden state h 3 \mathbf{h}_3 of time step 3${\mathbf{h}}_3$The calculation depends on the model parameters ${\mathbf{W}}_{hx}$和$W_{hh}$, and the hidden state $h_2$And the input x 3 \mathbf{x}_3 of the current time step${\mathbf{x}}_3$。

The figure above represents a computational graph of the dependencies of a recurrent neural network model with three time steps. Uncolored boxes represent variables, colored boxes represent parameters, and circles represent operators

As just said, the model parameters in the above figure are $W_{hx}$、$W_{hh}$和$W_{qh}$. Typically, training this model requires gradient calculations for these parameters: $\partial L/\partial {\mathbf{W}}_{hx}$ 、 $\partial L/\partial {\mathbf{W}}_{hh}$和$\partial L/\partial W_{qh}$. According to the dependencies in the above figure, we can traverse the calculation graph in the opposite direction of the arrow, and calculate and store gradients in turn. In order to flexibly express the multiplication of matrices, vectors and scalars of different shapes in the chain rule, we continue to use the $\text{prod}$ operator.

First, at any time step $t$ , the objective function to differentiate with respect to the model output is fairly straightforward:

$$\frac{\partial L}{\partial {\mathbf{o}}_t}=\frac{\partial l({\mathbf{o}}_t,y_t)}{T\cdot \partial o_t}\in {\mathbb{R}}^q.$$

Now, we can calculate the objective function with respect to the parameter ${\mathbf{W}}_{qh}$的梯度：$\partial L/\partial {\mathbf{W}}_{qh}\in R^{q\times h}$ . Based on the above figure, the objective function$L$通过${\mathbf{o}}_1,\dots ,o_T$depends on $W_{qh}$。 依据链式法则，得到
$$\frac{\partial L}{\partial {\mathbf{W}}_{qh}}=\sum _{t=1}^T\text{prod}\left(\frac{\partial L}{\partial {\mathbf{o}}_t},\frac{\partial o_t}{\partial W_{qh}}\right)=\sum _{t=1}^T\frac{\partial L}{\partial o_t}{\mathbf{h}}_t^{\top },$$
and next, as shown, at the final time step$T$ , the objective function$L$ only passes${\mathbf{o}}_T$Depends on the hidden state $h_T$. Therefore, we can easily get the gradient $\partial L/\partial {\mathbf{h}}_T\in {\mathbb{R}}^h$：
$$\frac{\partial L}{\partial {\mathbf{h}}_T}=\text{prod}\left(\frac{\partial L}{\partial {\mathbf{o}}_T},\frac{\partial o_T}{\partial h_T}\right)={\mathbf{W}}_{qh}^{\top }\frac{\partial L}{\partial o_T}.When$$
the objective function$L$ by${\mathbf{h}}_{t+1}$和${\mathbf{o}}_t$depends on ${\mathbf{h}}_t$, for any time step $t<T$ is getting trickier. According to the chain rule, the gradient of the hidden state$\partial L/\partial {\mathbf{h}}_t\in {\mathbb{R}}^h$ at any time step$t<T$时都可以递归地计算为：
$$\frac{\partial L}{\partial {\mathbf{h}}_t}=\text{prod}\left(\frac{\partial L}{\partial h_{t+1}},\frac{\partial h_{t+1}}{\partial h_t}\right)+\text{prod}\left(\frac{\partial L}{\partial {\mathbf{o}}_t},\frac{\partial o_t}{\partial h_t}\right)={\mathbf{W}}_{hh}^{\top }\frac{\partial L}{\partial h_{t+1}}+W_{qh}^{\top }\frac{\partial L}{\partial o_t}.$$

For analysis, for any time step $1\le t\le T$展开递归计算得
$$\frac{\partial L}{\partial {\mathbf{h}}_t}=\sum _{i=t}^T{\left({\mathbf{W}}_{hh}^{\top }\right)}^{T-i}{\mathbf{W}}_{qh}^{\top }\frac{\partial L}{\partial {\mathbf{o}}_{T+t-i}}.$$

We can see from the above formula that this simple linear example already exhibits some key problems of long sequence models: it gets stuck in ${\mathbf{W}}_{hh}^{\top }$A potentially very large power of . In this power, eigenvalues ​​less than 1 will disappear, and eigenvalues ​​greater than 1 will diverge. This is numerically unstable in the form of vanishing or exploding gradients. One way to solve this problem is to truncate the size of the time step (as mentioned in the first part) according to the computational convenience. In practice, this truncation is achieved by separating the gradient after a given number of time steps. Later, we will learn how more complex sequence models such as long short-term memory models further alleviate this problem.

Finally, the above figure shows that: the objective function $L$ passes through the hidden state${\mathbf{h}}_1,\dots ,h_T$Depends on the model parameters W hx \mathbf{W}_{hx} in the hidden layer${\mathbf{W}}_{hx}$和$W_{hh}$. To compute the gradient $\partial L/\partial {\mathbf{W}}_{hx}\in {\mathbb{R}}^{h\times d}$和$\partial L/\partial {\mathbf{W}}_{hh}\in {\mathbb{R}}^{h\times h}$， 我们应用链式规则得：
$$\begin{array}{r}\begin{array}{rl}\frac{\partial L}{\partial {\mathbf{W}}_{hx}}& =\sum _{t=1}^T\text{prod}\left(\frac{\partial L}{\partial {\mathbf{h}}_t},\frac{\partial h_t}{\partial W_{hx}}\right)=\sum _{t=1}^T\frac{\partial L}{\partial h_t}{\mathbf{x}}_t^{\top },\\ \frac{\partial L}{\partial W_{hh}}& =\sum _{t=1}^T\text{prod}\left(\frac{\partial L}{\partial h_t},\frac{\partial h_t}{\partial W_{hh}}\right)=\sum _{t=1}^T\frac{\partial L}{\partial h_t}{\mathbf{h}}_{t-1}^{\top },\end{array}\end{array}$$

其中$\partial L/\partial {\mathbf{h}}_t$是由$\frac{\partial L}{\partial h_T}=\text{prod}\left(\frac{\partial L}{\partial {\mathbf{o}}_T},\frac{\partial o_T}{\partial h_T}\right)={\mathbf{W}}_{qh}^{\top }\frac{\partial L}{\partial o_T}.$和$\frac{\partial L}{\partial {\mathbf{h}}_t}=\text{prod}\left(\frac{\partial L}{\partial h_{t+1}},\frac{\partial h_{t+1}}{\partial h_t}\right)+\text{prod}\left(\frac{\partial L}{\partial {\mathbf{o}}_t},\frac{\partial o_t}{\partial h_t}\right)={\mathbf{W}}_{hh}^{\top }\frac{\partial L}{\partial h_{t+1}}+W_{qh}^{\top }\frac{\partial L}{\partial o_t}.$ Obtained by recursive calculation, is a key quantity affecting numerical stability.

As we explained earlier, since backpropagation through time is how backpropagation is applied in recurrent neural networks, training recurrent neural networks uses forward propagation and backpropagation through time alternately. The above gradients are sequentially computed and stored through time backpropagation. Specifically, stored intermediate values ​​are reused to avoid double calculations, e.g. storing $\partial L/\partial {\mathbf{h}}_t$, so that when calculating $\partial L/\partial {\mathbf{W}}_{hx}$和$\partial L/\partial W_{hh}$used when.

3 Summary

• "Backpropagation through time" only applies to backpropagation on sequence models with hidden states.

• The truncation is a requirement for computational convenience and numerical stability. Truncation includes: regular truncation and random truncation.

• The high power of the matrix may cause the divergence or disappearance of the eigenvalues ​​of the neural network, which will be expressed in the form of gradient explosion or gradient disappearance.

• For computational efficiency, Backpropagation Through Time caches intermediate values ​​during computation.