Deep learning - recurrent neural network notes

　　Recurrent Neural Network (Recurrent Neural Network, RNN) and recurrent neural networks (Recursive Neural Network, RNN) on shorthand is the same, this note only for recurrent neural networks, Let me talk about this before, for these two neural networks some blog just its separate it from the name, there is no specific detail, after a lot of Chinese search, still found there are a lot of enthusiastic great God tells them carefully, we recommend temporarily https: // zybuluo. com / hanbingtao / note / 541458 this description. Most of this switched notes the article.

　　Fully connected neural network and neural network convolution process they can only enter one by one separately, after a previous input and the input is totally irrelevant. However, some tasks need to be able better information processing sequence, i.e., the input and the input front of the latter there is a relationship. For example, when we understand the meaning of a sentence, isolated understand every word of this sentence is not enough, we need to deal with the entire sequence of these words are connected together; when we deal with the video, we can not just go alone analyzes each frame, and to analyze the entire sequence of the frames are connected together. At this point, you need to use the depth of field of study is important in another type of neural networks: recurrent neural network (Recurrent Neural Network).

　　Below is a simple loop such as neural network, which consists of an input layer, a hidden layer and an output layer consisting of:

　　If the above that there is a circle with an arrow W removed, it becomes the most common fully connected neural network. x is a vector that represents the value of the input layer (there is no painted circle that represents neuron node); s is a vector that represents the value of the hidden layer (here drew a hidden layer of nodes, you can imagine this one is actually a plurality of nodes, the same number of nodes and the vector s dimension); the U-is the right input layer to the hidden layer weight matrix; O is a vector that represents the output layer; V is the weight of the hidden layer to the output layer weight matrix. Value s hidden layer of the neural network cycle depends not only on the current input x, s time also depends on the value of the hidden layer. W is a weight matrix of hidden layer as a value on the right to re-enter this time.

　　If the top of a development, recurrent neural network can also be painted look like this:

The network input is received at time t X _t

Formula 1 formula is the output layer, the output layer is a layer fully connected, i.e. each of its nodes and each node is connected to the hidden layer. V is the output layer weight matrix, g is the activation function. Equation 2 is calculated in the hidden layer, which layer is cyclic. U is the input x weight weight matrix, W is the last value S _{T-. 1}

Output value ot about S _T expression, which is input by the foregoing previous value X _{T-. 1} , X _T-2 , X _{T-. 3} , ... effects, which is why any cycle neural network can look forward a plurality of input values reasons.

Bidirectional Recurrent Neural Networks

FIG bidirectional loop network:

Consider first FIG. Y ₂ calculated, it can be seen from the figure, the bidirectional convolutional neural network hidden layer to store two values, a positive A participating calculated, another value A 'participation reverse calculation. The final output value Y ₂ depends on the A ₂ and A ' ₂ , can be calculated as follows:

General rule can be seen: the forward calculation, the value of s hidden layer _t

We can see from the above three formulas to calculate forward and reverse calculations do not share the weight, that is to say U and U ', W and W', V and V 'are different weight matrix.

Depth Recurrent Neural Networks

　　Recurrent Neural Networks described earlier only one hidden layer, of course, also be stacked two or more hidden layers, thus obtaining a depth of Recurrent Neural Networks. As shown below:

Recurrent Neural Network Training

Neural network training algorithm cycle: BPTT

BPTT algorithm for training algorithm circulation layer, its basic principles and BP algorithm is the same, also contain the same three steps:

Each neuron is calculated before the output value;
Δ reverse calculation error term of each neuron _j
Calculate the weight of each weight gradient.

Finally then stochastic gradient descent algorithm updates the weights. Since https://zybuluo.com/hanbingtao/note/541458 This use of the numerical calculations inside the matrix operation approach, a long time a little bit forgotten, after much searching found https://www.jianshu.com/p / 87aa03352eb9 mathematical expression of this easier to understand. Layer loop as shown below:

上图表明了RNN网络的完整拓扑结构，从图中我们可以看到RNN网络中的参数情况。在这里我们只分析t时刻网络的行为与数学推导。t时刻网络迎来一个输入x_t，网络此时刻的神经元状态s_t用如下式子表达：

为了方便做了如下变换：

RNN的损失函数选用交叉熵（Cross Entropy），这是机器学习中使用最广泛的损失函数之一了，其通常的表达式如下所示：

上面式子是交叉熵的标量形式，y _i是真实的标签值，y ^* _i是模型给出的预测值，最外面之所以有一个累加符号是因为模型输出的一般都是一个多维的向量，只有把n维损失都加和才能得到真实的损失值。交叉熵在应用于RNN时需要做一些改变：首先，RNN的输出是向量形式，没有必要将所有维度都加在一起，直接把损失值用向量表达就可以了；其次，由于RNN模型处理的是序列问题，因此其模型损失不能只是一个时刻的损失，应该包含全部N个时刻的损失。

故RNN模型在 t时刻的损失函数写成：

全部N个时刻的损失函数（全局损失）表达为如下形式：

需要说明的是：y_t是t时刻输入的真实标签值，o_t为模型的预测值，N代表全部N个时刻。下文中为了书写方便，将Loss简记为L。在结束本小节之前，最后补充一个softmax函数的求导公式：

BPTT算法

由于RNN模型与时间序列有关，因此不能直接使用BP（back propagation）算法。针对RNN问题的特殊情况，提出了BPTT算法。BPTT的全称是“随时间变化的反向传播算法”（back propagation through time）。这个方法的基础仍然是常规的链式求导法则，接下来开始具体推导。虽然RNN的全局损失是与全部N个时刻有关的，但为了简单笔者在推导时只关注t时刻的损失函数。

首先求出t时刻下损失函数关于o_t^*的微分：