After reading a lot of blogs, I found a usage self.v = torch.nn.Parameter (torch.FloatTensor (hidden_size)), first of all, I can understand this function as a type conversion function and convert an untrainable type Tensor into Training type parameter and bind this parameter to this module (net.parameter () has this bound parameter, so it can be optimized when the parameter is optimized), so after the type conversion, this self.v becomes It became part of the model and became a parameter that can be changed in the model according to training. The purpose of using this function is also to allow some variables to continuously modify their values in the process of learning to achieve optimization.
Where this function appears
In the concat attention mechanism, the weight V is constantly learned, so if the parameter type, not directly using a torch.nn.Linear () may be because the learning effect is not good.
Through the following experiments, it was found that the weight and bias in linear are the parameter type, and cannot be replaced by the tensor type, and the weight in linear may even change the model by specifying a shape different from the initialization.
self.gamma is bound to the model, so it can be optimized during training.
