RNN(Recurrent Neural Network)
Why do you need RNN?
For example, one task requires identifying the meaning of each word. In the following sentences, taipei represents the destination and departure place respectively. We hope that the neural network can identify different semantics in different sentences. However, for the previous feedforward neural network, if the same word is input, the output will be different. will be the same, it cannot say under different circumstances that Taipei is more likely to be the destination, or that it is more likely to be the departure place.
But if we judge, we know the semantics of taipei through arrive and leave, so we also hope that our neural network can remember the previous word
For example, when entering the first sentence, when we enter Taipei, we know that the previous word is arrival. The second sentence is the same is used to identify different semantics of the same word.
At this time RNN was introduced, our neurons have memory
For the neurons in each layer, when the neural network has input, each layer Example: (that is, the value output to the next layer of neurons) when there is another input. The neurons in each layer already have the previous calculation value, and use this value to calculate the next neuron. (For the first output, each neuron has an initial value)The neuron will save the currently calculated value
The following figure shows a fully connected neural network. There are three input sequences. Assume that the weights are all 1 and there is no bias.For the first sequence [1, 1] is different from the output of the second sequence [1,1], because when input for the first time, the hidden layer neuron saves the initial value 0, but the output of the second sequence [1,1] is different. The value of the second input neuron is the value calculated for the first time, which is [2,2], soeven if it is the same input, different results will be obtained
< a i=4>. And we can also find that if changes the order of the inputs, their output will also change, so RNN will consider the order of the inputs < a i=7>.
Therefore, the same is true for the previous sentences. Although Taipei is input, the values saved in the hidden layer are different, so if Taipei is input equally, the output will be different.
For information about GRU, please refer to our blog (Ng Enda) - Deep Learning - GRU Unit
For information about LSTM, please refer to Our blog (Wenda) - Deep Learning - LSTM
