[PyTorch] rnn, lstm, gru dimension input output

In this article refers to RNN LSTM, GRU, and so on
CNN and RNN in batchSize default location is different.

• CNN in : batchsize position position 0.
• In RNN : the BatchSize position position 1.

Input Data Format In RNN:

For the simplest RNN, we can use two ways to call torch.nn.RNNCell(), it only accepts the sequence of single-step input, you must explicitly incoming hidden . torch.nn.RNN()Can accept a sequence input by default will pass a hidden state of all zeros , you can declare your own hidden pass.

1. Input size is a three-dimensional tensor[seq_len,batch_size,input_dim]

• input_dimIs the dimension of input, such as a128
• batch_sizeTo be a RNNnumber of input sentences, such as is 5.
• seq_lenIs the maximum length of a sentence, for example, 15
  so do note that RNNthe input is a sequence, a batch of all sentences are entered, the resulting ouptutand hiddenall of the output and hidden all this batch, the dimension is three-dimensional.
  ** now can be understood as a total batch_sizeseparate RNNcomponents, RNNthe dimension is input input_dim, the input total seq_lentime steps, each time step input to this whole RNNdimension of the module is[batch_size,input_dim]

# 构造RNN网络，x的维度5，隐层的维度10,网络的层数2
rnn_seq = nn.RNN(5, 10,2)  
# 构造一个输入序列，句长为 6，batch 是 3， 每个单词使用长度是 5的向量表示 x = torch.randn(6, 3, 5) #out,ht = rnn_seq(x,h0) out,ht = rnn_seq(x) #h0可以指定或者不指定 

Question 1 : Here out, htthe size is how much?
Answer : : out3 * 6 * 10 ht: 10 * 2 * 3, outoutput dimensions [seq_len,batch_size,output_dim], ht dimension [num_layers * num_directions, batch, hidden_size], if unidirectional monolayers of a sentence is only then RNN ahidden .
Question 2 : out[-1]and ht[-1]are equal?
Answer : equal, hidden unit is the last unit of output, it is conceivable, in fact, that the output of each time step of the hidden units

2. RNNOther parameters

RNN(input_dim ,hidden_dim ,num_layers ，…)
– input_dim 表示输入的特征维度
– hidden_dim 表示输出的特征维度，如果没有特殊变化，相当于out
– num_layers 表示网络的层数
– nonlinearity 表示选用的非线性激活函数，默认是 ‘tanh’
– bias 表示是否使用偏置，默认使用
– batch_first 表示输入数据的形式，默认是 False，就是这样形式，(seq, batch, feature)，也就是将序列长度放在第一位，batch 放在第二位
– dropout 表示是否在输出层应用 dropout
– bidirectional 表示是否使用双向的 rnn，默认是 False

 

In the shape of the input to the RNN tensor

LSTM output a plurality of memory cells

# 输入维度 50，隐层100维，两层
lstm_seq = nn.LSTM(50, 100, num_layers=2)
# 输入序列seq= 10，batch =3，输入维度=50 lstm_input = torch.randn(10, 3, 50) out, (h, c) = lstm_seq(lstm_input) # 使用默认的全 0 隐藏状态 

Question 1 : outand (h,c)the size is the number?
Answer : out: (3 * 10 * 100), (h,c): is (100 * 2 * 3)
Question 2 : out[-1,:,:]and h[-1,:,:]equal to it?
Answer : equal

GRU more like a traditional RNN

gru_seq = nn.GRU(10, 20,2) # x_dim,h_dim,layer_num
gru_input = torch.randn(3, 32, 10) # seq，batch，x_dim out, h = gru_seq(gru_input)

Author: VanJordan
link: https: //www.jianshu.com/p/b942e65cb0a3
Source: Jane book
Jane book copyright reserved by the authors, are reproduced in any form, please contact the author to obtain authorization and indicate the source.