PyTorch framework
some usage
underscore suffix
In PyTorch, functions with an underscore suffix usually indicate an inplace operation. In-place modification operations modify the original tensor object directly without returning a new tensor. This saves memory and computing resources when working with large data sets.
For example, scatter_() is an in-place modification operation used for scatter operations. It directly modifies the original target tensor without returning a new tensor. Similarly, functions like add_(), mul_(), div_() are also in-place modification operations for in-place addition, in-place multiplication, in-place division, etc.
However, not all functions have an underscore suffix. Many functions simply return new tensors without modifying the original tensors. For example, functions like add(), mul(), div(), scatter(), etc. are immutable operations that create and return a new tensor object.
Note that not all functions have in-place modified versions. Some functions only have versions for immutable operations and no corresponding in-place modification versions.
Therefore, when you need to modify a tensor in place, you can look for the corresponding function with an underscore suffix. But not all functions have this suffix. When using a function, you need to check the function's documentation or refer to the official documentation to determine whether there is an in-place modified version.
The difference between torch.nn.conv2d and torch.nn.functional.conv2d
We will find that there are some functionally repeated operations in torch.nn.xxx and torch.nn.functional.xxx, such as convolution, activation, pooling. How are these operations different? What is the use of each?
The layer implemented by nn.Module is a special class defined by class Layer(nn.Module).
The functions in nn.functional are more like pure functions, defined by def function(input)
The explanation given by chat
In PyTorch, torch.nn and torch.nn.functional provide some repetitive operations, such as convolution, activation, and pooling. The differences between these operations mainly focus on usage and functions. The following are their main differences and uses:
1. How to use: The operations in torch.nn are implemented by creating corresponding layer objects, which are subclasses of torch.nn.Module and need to be instantiated and used as objects . The operation in torch.nn.functional is a way to directly call the function . There is no concept of layer object, and the input and parameters can be directly passed in for calculation.
2. Calculation graph recording: The operation of torch.nn will automatically track the computation graph and record the information of layer objects and parameters for automatic derivation and gradient calculation. However, the operation of torch.nn.functional will not record the calculation graph, cannot perform automatic derivation, and can only calculate the gradient manually.
3. Flexibility: torch.nn.functional provides a more flexible programming method, and the input and parameters of the function can be specified and adjusted more flexibly. It is suitable for those cases where parameters need to be adjusted dynamically according to the input. However, when the operations in torch.nn instantiate the layer object, the parameters need to be fixed, which limits the flexibility and is suitable for those cases where the parameters are fixed.
4. Reusability: Layer objects in torch.nn are reusable and can be used multiple times in different models. This makes it easy to build complex model structures. The function of torch.nn.functional is absolutely stateless, does not save any parameters, and needs to specify the value of the parameter every time it is called. **As such, they are not reusable and cannot be shared across multiple places.
To sum up, torch.nn and torch.nn.functional provide different programming methods and functions, and you can choose the appropriate method according to your specific needs. If you need to keep track of computational graphs, do automatic differentiation, and build reusable model structures, it is recommended to use operations in torch.nn. If you need a more flexible programming method and increase the readability of the code, or operate in an environment that does not require calculation graphs and automatic derivation, you can use the functions in torch.nn.functional.
function
torch.function name
The PyTorch framework provides many useful functions, which can be called by torch.function name. The following are some commonly used torch. functions:
1.torch.max: Calculate the maximum value of a tensor in a specified dimension or return the maximum value of each position in two tensors.
2.torch.min: Calculate the minimum value of a tensor in a specified dimension or return the minimum value of each position in two tensors.
3.torch.mean: Calculate the mean value of the tensor in the specified dimension.
4.torch.sum: Calculate the sum of tensors along the specified dimension.
5.torch.prod: Calculate the product of tensors in the specified dimension.
6.torch.argmax: Returns the index of the maximum value of the tensor in the specified dimension.
7.torch.argmin: Returns the index of the minimum value of the tensor in the specified dimension.
8.torch.exp: Calculate the exponent of the tensor.
9.torch.log: Calculate the natural logarithm of the tensor.
10.torch.sqrt: Calculate the square root of a tensor.
11.torch.abs: Calculate the absolute value of the tensor.
12.torch.sin: Calculate the sine of the tensor.
13.torch.cos: Calculate the cosine of the tensor.
14.torch.matmul: Perform matrix multiplication of two tensors.
15.torch.cat: Concatenate tensors along the specified dimension.
16.torch.split: Split tensor by specified size or quantity.
17.torch.reshape: Change the shape of the tensor.
This is just a small sample, PyTorch provides many other useful functions, covering functions for various tensor operations, mathematical operations, and neural networks. You can refer to the official PyTorch documentation for a complete list of functions and detailed usage instructions.
scatter function
some cases
import torch
input = torch.zeros(2, 3) # 创建一个2x3的全零张量
index = torch.tensor([[0, 1, 0], [1, 0, 1]]) # 索引张量
src = torch.tensor([[1, 2, 3], [4, 5, 6]],dtype=input.dtype) # 要散布的值
output = input.scatter(dim=0, index=index, src=src)
print(output)
#output
tensor([[1., 5., 3.],
[4., 2., 6.]])
Personal understanding:
scatter is a function in PyTorch that is used to scatter the specified value to the specified position of the target tensor. Its main function is to fill the specified value into the corresponding position of the target tensor according to the given index.
torch.scatter(input, dim, index, src)
- input: A target tensor to receive the scatter results.
- dim: An integer value specifying the dimension of the index.
- index: An index tensor specifying where to scatter values.
- src: A source tensor containing the values to scatter.
The implementation process can be easily understood in this way
traverse src
for i in range(src.shape[0]):
for j in range(src.shape[1]):
for k in range(src.shape[2]):
input[index[i][j][k]][j][k] = src[i][j][k] # if dim == 0
input[i][index[i][j][k]][k] = src[i][j][k] # if dim == 1
input[i][j][index[i][j][k]] = src[i][j][k] # if dim == 2
input[index[i][j][k]][j][k] = src[i][j][k] # if dim == 0
input[i][index[i][j][k]][k] = src[i][j][k] # if dim == 1
input[i][j][index[i][j][k]] = src[i][j][k] # if dim == 2
other
Different modes of the model: train, eval, infer
1. Training mode (train): This is the default mode of the model. In training mode, the model performs a complete training process, including forward propagation, calculating loss, backpropagation, and parameter update. In this mode, the model will enable training-related operations such as Dropout and Batch Normalization to better adapt to the training data.
2. Evaluation mode (eval): In evaluation mode, the model turns off some operations with randomness , such as random discarding of Dropout and Batch Normalization. The purpose of this is to obtain consistent output during the evaluation phase and to ensure that the model's results are reproducible. In evaluation mode, the forward propagation behavior of the model may differ from training mode.
This method is generally used with the model's torch.no_grad() context manager to avoid unnecessary gradient calculations and improve evaluation efficiency
example
model.eval()
with torch.no_grad():
# 进行评估操作，不计算梯度
# ...
3. Inference mode (infer): The inference mode usually refers to the mode in which the model makes predictions on new data in real applications. In this mode, the model is similar to the evaluation mode, and some training-related operations are turned off to maintain output stability. Inference patterns are conceptually similar to evaluation patterns and can sometimes be used to express the same behavior.
The switching of these modes can be realized by calling the corresponding methods of the model object, such as model.train(), model.eval(). The purpose of switching modes is to adjust the behavior of the model according to the different stages or requirements of the task to obtain the best training results or prediction performance. The training mode is usually used in the training phase, while the evaluation mode or inference mode is usually used in the evaluation and inference phases.
Read and save tensor
import torch
tensor=torch.tensor([1,2,3,4,5])
tensor.save(tensor,"tensor.pt")
tensor_load=tensor.load("tensor.pt")