RNN neural network

I. Overview

Recurrent Neural Network (RNN) is a type of recursive neural network that takes sequence data as input, performs recursion in the evolution direction of the sequence , and has all nodes (cyclic units) connected in a chain. network), which captures the pattern characteristics between sequences through the internal structure of the network, and is generally output in the form of a sequence.

RNN (Recurrent Neural Network) is a neural network with recurrence.

Suppose we have a set of vectors $(x^{(1)},x^{(2)},...,x^{(n)})$ , explained by the hypothetical function $H$

$H^{(t)}=\sigma(W^{ht}\cdot X{(t)}+W^{hh}\cdot H^{(t-1)}+b_n)$

$t$ Represents the amount of time and $\sigma$ is a nonlinear function, $Y^{(t)}$ corresponding to the output of each cycle. We often take the last output in continuous time series operations.

The expansion of the loop process represents the hierarchical depth of the RNN network.

RNN is derived from feed-forward neural networks and can utilize its internal state (memory) to process variable-length input sequences. This makes RNN more suitable for tasks such as unsegmented, continuous handwriting recognition or speech recognition.

RNN workflow

The vector $(x^{(1)},x^{(2)},...,x^{(n)})$ imports an element each time through the loop, and the model transfers internal state $H_{t-1}$ from one cell to the next in time. Note that all cells use the same weight $W$ .

RNN in PyTorch

The complete RNN in pytorch torch.nn.RNN is implemented through classes.

RNN — PyTorch 1.12 documentation

parameter:

input_size – the number of features of the input sample x

hidden_size – number of features in the hidden state

num_layers – Number of loop layers. For example, setting num_layers=2means stacking two RNNs together to form a stacked RNN, where the second RNN receives the output of the first RNN and computes the final result. The default value is 1

nonlinearity – the nonlinear function to use. Can be "tanh"or "relu". default value:'tanh'

bias – If False, this layer does not use bias weights $b_{ih}$ and $b_{hh}$. default value:True

batch_first – If True, the input and output tensors will be as (batch, seq, feature) instead of (seq, batch, feature) . Note that this does not apply to hidden or cell states. default value:False

dropout – If non-zero, introduce a dropout layer on the output of each RNN layer except the last layer, with dropout probability equal to dropout. Default value: 0

bidirectional – if True, use a bidirectional RNN. default value:False

Pytorch code implementation

class RNN_Classfication(nn.Module):
    def __init__(self, input_size, hidden_size, num_layers, num_class):
        super(RNN_Classfication, self).__init__()
 
        self.rnn = nn.RNN(        # RNN模型
            input_size = input_size,      # 图片每行的数据像素点(特征数)
            hidden_size = hidden_size,     # rnn 隐藏层单元数
            num_layers = 1,       # 有几层 RNN layers
            batch_first = True,   # 指定batch为第一维 e.g. (batch, time_step, input_size)
        )
        self.out = nn.Linear(hidden_size, num_class)    # 输出层
 
    def forward(self, x):
        # x shape (batch, time_step, input_size)
        # r_out shape (batch, time_step, output_size)
        # h_n shape (batch, hidden_size) rnn hidden
        r_out, h_n = self.rnn(x)   
        
        # 选取最后一个时间点的 r_out 输出
        # 这里 r_out[:, -1, :] 的值也是 h_n 的值
        out = self.out(r_out[:,-1,:])
        # out = self.out(h_n.squeeze(0))
        return out

pytorch implements RNN recurrent neural network

RNN neural network

RNN in PyTorch

Guess you like