Speed pass pytorch library (long text)
foreword
This article is mainly prepared for those friends who are not familiar with the pytorch library and have not yet started. It sorts out the main content of the pytorch library and helps you get started with one of the most important libraries for deep learning.
Directory Structure
Article directory
-
- Speed pass pytorch library (long text)
-
- 1. Introduction and installation
- 2. Tensor and its common methods:
- 3. Automatic derivation:
- 4. Model creation and definition:
- 5. Data loading and reading:
- 6. Model training and evaluation:
- 7. Visualization:
- 8. Call the GPU:
- 9. torchvision module:
- 10. Model saving and loading:
- 9. torchvision module:
- 10. Model saving and loading:
1. Introduction and installation
1.1 Introduction
The pytorch library is currently one of the most used deep learning libraries, and the other is tensorflow. Regarding the choice of the two, first of all, it depends on what framework your research group/company uses. If you are a free learner, you can choose to learn both, but the recommendation is pytorch.
1.2 Installation
There are already many detailed installation tutorials on the Internet for pytorch installation, and you can search for them yourself. However, it should be noted that the GPU version must be installed, otherwise many projects will report errors when running.
2. Tensor and its common methods:
Tensor, or tensor, is the basic operation object in PyTorch, which can be regarded as a multidimensional matrix containing elements of a single data type. From the perspective of use, Tensor is very similar to NumPy's ndarrays, and they can be freely converted between each other , except that Tensor also supports GPU acceleration.
Therefore, tensor can be boldly regarded as an array-like object.
2.1 Create Tensor:
common creation
a = torch.tensor([[1,2],
[3,4]])
print(a)
Note : If the data are all integers, tensor defaults to INT64 type, and once there are decimals, the type will change to float32.
Create a tensor with zero data
# 创建数据为零的张量
b = torch.zeros((2,3)) # 输入指定形状即可
print(b) # 默认为float32
Create a tensor with data 1
# 创建数据为1的张量
c = torch.ones((3,3,3))
print(c)
Create the identity matrix
# 创建单位矩阵
d = torch.eye(3)
print(d)
Create a tensor whose data is random values
# 创建数据为随机值的张量
e = torch.randn((2,3)) # 正态分布，均值为0，方差为1
f = torch.rand((2,3)) # 均匀分布，[0,1)的均匀分布
print(e)
print(f)
Create a 1D tensor of the specified range
g = torch.arange(1,20,3) # 起始值，结束值，步长
print(g)
h = torch.linspace(1,100,100) # [start=1,end-100]个数100个
print(h)
print(h.size())
2.2 Tensor object properties:
shape
a = torch.ones((3,4))
print(a.shape)
print(a.size())
type of data
a = torch.ones((3,4))
print(a.dtype)
Method: numel()
# 返回元素个数
a = torch.ones((3,4))
print(a.numel()) # 12
2.3 Common operations:
| operation | method | parameter | illustrate |
|---|---|---|---|
| absolute value | torch.abs(a) | ||
| add up | + or torch.add(c,d) | ||
| cut out | torch.clamp(e,2,4) | clipping object, lower bound, upper bound | Compare each value in the tensor with the upper bound and lower bound, if it is lower than the lower bound, it will be rewritten as the lower bound value, if it is higher than the upper bound, it will be rewritten as the upper bound value, and the value in the middle will remain unchanged . |
| division | torch.div(f,2) | Division is the division of corresponding elements | |
| exponentiation | torch.pow(h,3) | ||
| Multiply corresponding elements | torch.mul(f,g) | ||
| matrix multiplication | torch.mm(i,j) | ||
| Judgment size | torch.ge(input, other) | the tensor to compare, the tensor to compare | The values of the two tensors are compared one by one, and the position with the same value is 1, otherwise it is 0. Similar methods are: le ( whether less than or equal to ), ge ( whether greater than or equal to ) |
| get index position | torch.where(condition，a，b) | condition is a conditional judgment. When the condition is met, the corresponding position is the value of a, otherwise it is the value of b | Generally used in combination with the above method, for example: torch .where(torch.eq(a,b[:,None])) |
2.4 tensor to array conversion:
tensor —> array
a = torch.tensor([2,3])
b = a.numpy()
c = np.array(a)
print(type(b))
print(type(c))
array—>tensor
d = np.array([2,3])
e = torch.tensor(d)
print(type(e))
2.5 Index operation:
There is no difference between the tensor index and the ordinary index, but the memory sharing in the tensor should be noted :
import torch
a = torch.randn((3,4))
b = a[0,0]
print(b)
b += 1
print(a)
The result of the operation is:
tensor(-0.4613)
tensor([[ 0.5387, -1.2455, 0.4596, -0.9547],
[ 1.7746, 0.2623, -0.2514, 1.4591],
[-0.1769, 2.2323, 0.1277, 0.2700]])
That is , although we have assigned a[0,0] to the variable b, since both memory points to the same address, if you modify b, the corresponding value of a will also change.
2.6 Shape change:
import torch
a = torch.randn(5,4)
# 先克隆一个
b = torch.clone(a)
# 再改变形状
c = a.view(4,5)
print(c)
Note : Due to memory sharing, before changing the shape, clone a backup first, and then change the shape.
There is also a similar method:
import torch
a = torch.randn(5,4)
b = torch.reshape(a,(4,5)) # 参照numpy方法，不过view只能用于连续内存
print(b)
There is also a very common form of the above
a.view(-1,5) # 表示第二个维度为5，第一个维度的值自己推导
2.7 Splicing:
| method | parameter |
|---|---|
| torch.cat() | The first parameter, (A,B,…), is the object to be spliced. The second parameter, dim, specifies which dimension to splice from. 0 means splicing by row (the splicing dimensions must be the same) |
| torch.stack() | The first parameter, (a,b), is the object to be spliced, and the shape must be the same. The second parameter, dim, is the index of the newly added dimension after splicing |
Example:
import torch
a = torch.tensor([[1, 2, 3], [4, 5, 6], [7, 8, 9]])
b = torch.tensor([[11, 22, 33], [44, 55, 66], [77, 88, 99]])
c = torch.stack([a, b], 0)
print(a)
print(b)
print(c)
# 输出信息：
tensor([[1, 2, 3],
[4, 5, 6],
[7, 8, 9]])
tensor([[11, 22, 33],
[44, 55, 66],
[77, 88, 99]])
tensor([[[ 1, 2, 3],
[ 4, 5, 6],
[ 7, 8, 9]],
[[11, 22, 33],
[44, 55, 66],
[77, 88, 99]]])
2.8 Summary of other commonly used methods:
| method name | effect | Example/parameter explanation |
|---|---|---|
| squeeze() | Compress the dimensions of the data and remove the dimensions with a dimension of 1 | squeeze(0): Represents removing the first dimension if the value of the first dimension is 1 |
| unsqueeze() | Expand the dimension of the data, that is, increase the dimension | x.unsqueeze(0), that is, expand a dimension in the first dimension |
| sort() | Sort the elements of tensor according to the specified dimension | Parameters: descending(bool, optional): False indicates ascending order, True indicates descending order Return value: the first tensor indicates the sorting of tensor values of the corresponding dimension; the second tensor indicates the index sequence of tensor values of the corresponding dimension |
| name() | Quickly check how many elements a tensor has |
3. Automatic derivation:
Automatic derivation is the core function of pytorch, and it is also the core of our implementation of our neural network algorithm. Its main application is to deal with the backpropagation algorithm .
3.1 Automatic derivation:
basic method
Below, we demonstrate through an example:
构建 y = x + b，然后对y求导
Then, the implementation method is as follows:
import torch
b = torch.randn((3,4),requires_grad=True) # requires_grad = True 必须声明
x = torch.randn((3,4),requires_grad=True)
# 定义一个函数
t = x + b
y = t.sum() # 这一步是因为只有标量才可以求导（pytorch所规定）
# 求导
y.backward()
# b的梯度
print(b.grad)
# x的梯度
print(x.grad)
The result of the operation is as follows:
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
# 正确结果，因为为线性函数嘛
It is not difficult to see from the above that the steps required for automatic derivation are:
# 1. 定义数据和函数
xxxx = torch.tensor(xxx,requires_grad=True)
fx = xxxx
# 2. 对目标求导：fx要为标量
fx.backward()
# 3. 计算想要的参数的梯度
print(xx.grad)
Deep Dive 1
Want to explore: Do all variables have to declare requires_grad=True?
- All variables are declared
This is the example above, and it obviously works.
- only one variable declaration
Code one:
import torch
b = torch.randn((3,4),requires_grad=True)
x = torch.randn((3,4))
# 定义一个函数
t = x + b
y = t.sum()
# 求导
y.backward()
# b的梯度
print(b.grad)
# x的梯度
print(x.grad)
The result is:
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
None
Code 2:
import torch
b = torch.randn((3,4))
x = torch.randn((3,4),requires_grad=True)
# 定义一个函数
t = x + b
y = t.sum()
# 求导
y.backward()
# b的梯度
print(b.grad)
# x的梯度
print(x.grad)
The result is:
None
tensor([[1., 1., 1., 1.],
[1., 1., 1., 1.],
[1., 1., 1., 1.]])
- declare none
Obviously, this result will definitely report an error.
- Summarize
Through the above test, it can be seen that if the variable you want to solve for the gradient is the most basic (bottom, not composed of other variables) variable, then you must declare requires_grad=True for it, but if you are not interested in it, you can No need to set.
In-Depth Study 2
Another question: Are all variables declared requires_grad=True before the function can call the backward method?
This answer, in fact, the above case has told us - no. So, is it declaring a variable, so that the function composed of this variable can use the backward method?
Look at the code:
import torch
b = torch.randn((3,4))
x = torch.randn((3,4),requires_grad=True)
# 定义一个函数
t1 = x + b # 由x构成的函数
y1 = t1.sum()
t2 = b*3 # 不由x构成的函数
y2 = t2.sum()
# 查看是否可以
print(y1.requires_grad) # 返回变量requires_grad是否为True
print(y2.requires_grad)
The result is:
True
False
In summary
3.2 Control derivation:
Sometimes , some of our quantities do not need to participate in the gradient calculation . At this point, this method is needed:
import torch
x = torch.randn((3,4),requires_grad=True)
with torch.no_grad(): # 这样y不受x的requires_grad=True影响
y = x ** 2
print(y.requires_grad) #false
3.3 Clear the cumulative value:
The gradient calculation will accumulate by default, but in general we don't need to accumulate:
import torch
x = torch.tensor([10,20,30,40],requires_grad=True,dtype=torch.float32)
for i in range(5):
y = x**2 +10
# 转为标量
y1 = y.mean()
# 自动求导
y1.backward()
print(x.grad)
Running result:
tensor([ 5., 10., 15., 20.])
tensor([10., 20., 30., 40.])
tensor([15., 30., 45., 60.])
tensor([20., 40., 60., 80.])
tensor([ 25., 50., 75., 100.])
Therefore, it is necessary to clear the accumulated value:
import torch
x = torch.tensor([10,20,30,40],requires_grad=True,dtype=torch.float32)
for i in range(5):
y = x**2 +10
# 转为标量
y1 = y.mean()
# 清楚累加
if x.grad is not None:
x.grad.data.zero_()
# 自动求导
y1.backward()
print(x.grad)
3.4 Variable usage
FloatTensor When we created , there was no parameter specification requires_grad=True, because this FloatTensoris a class, not our commonly used method. If you want to add this type of variable to the differential system, it can be automatically derived, which requires special treatment.
method one
Use the Variable method:
import torch
from torch.autograd import Variable
b = torch.FloatTensor([1,2,3])
b = Variable(b)
print(b.requires_grad)
Method Two
Use built-in properties:
import torch
a = torch.FloatTensor([1,2,3]).requires_grad_() # 默认为True
print(a.requires_grad)
3.5 Other Notes:
1. 只有浮点型和复数型可以进行自动梯度求导
2. 自动求导默认累加，需要自己清空
3. 只有标量可以自动求导，因此需要手动转换，比如上面中t.sum()
4. 当一个张量设置了requires_grad=True后，不可以直接转为numpy对象，需要先使用detach方法生成一个新张量，然后进行转换。
Regarding the fourth point, here is an example:
import torch
x = torch.randn((3,4),requires_grad=True)
c = x.numpy()
If the above code is run directly, an error will be reported and needs to be modified to:
import torch
x = torch.randn((3,4),requires_grad=True)
new_x = x.detach()
c = new_x.numpy()
print(type(c)) # <class 'numpy.ndarray'>
4. Model creation and definition:
4.1 Methods involved in model building:
4.1.1 Activation function:
First of all, you need to know the name of each activation function, and then use it according to the following template:
import torch
from matplotlib import pyplot as plt
x = torch.arange(-100,100,1)
y = torch.sigmoid(x) # torch.函数名字
plt.figure()
plt.plot(x,y)
plt.show()
A small reminder: the above torch.xxx is the method of drawing functions, and torch.nn.xxxx is still used for specific use
# softmax多分类
torch.nn.LogSoftmax(dim=1/0)
# 1 表示按行求和为1
# 0 表示按列求和为1
4.1.2 Parameter initialization:
The main parameter methods are:
1. 均值初始化
2. 高斯分布初始化
3. Xavier初始化
4. He初始化
5. 其它初始化：固定值初始化等
Below, the relevant APIs are introduced one by one:
import torch
import torch.nn as nn
# 1. 均匀分布
def test1():
# 定义一个层，这样才有参数
linear = nn.Linear(5,3) # 输入特征维度为5，输出特征维度为3
nn.init.uniform_(linear.weight)
print(linear.weight)
# 2. 高斯分布
def test2():
# 定义一个层，这样才有参数
linear = nn.Linear(5,3) # 输入特征维度为5，输出特征维度为3
nn.init.normal_(linear.weight,mean=0,std=1)
print(linear.weight)
# 3. Xavier初始化
def test3():
# 定义一个层，这样才有参数
linear = nn.Linear(5,3) # 输入特征维度为5，输出特征维度为3
nn.init.xavier_normal_(linear.weight) # 可以为normal（高斯）、uniform（均匀）
print(linear.weight)
# 4. He初始化
def test4():
# 定义一个层，这样才有参数
linear = nn.Linear(5,3) # 输入特征维度为5，输出特征维度为3
nn.init.kaiming_uniform_(linear.weight)
print(linear.weight)
# 5. 固定值初始化
def test5():
# 定义一个层，这样才有参数
linear = nn.Linear(5,3) # 输入特征维度为5，输出特征维度为3
nn.init.constant_(linear.weight,5)
print(linear.weight)
4.1.3 Loss function:
Below, several commonly used loss functions are introduced, and other loss functions can be expanded.
The general template used is:
# 1. 创建损失对象
loss = nn.L1Loss()
# 2. 计算损失
output = loss(input, target)
# 3. 计算损失的梯度
output.backward()
It needs to be added: the loss object is zero-dimensional, so you can’t use the index to get the value, you can only get it through the item, although I don’t understand the principle.
loss_obj.item()
L1 loss function
Computes the absolute value of the difference between the output yand the true label .target
torch.nn.L1Loss(size_average=None, reduce=None, reduction='mean')
'''
参数：
reduction参数决定了计算模式。有三种计算模式可选：
none：逐个元素计算
sum：所有元素求和，返回标量。
mean：加权平均，返回标量
'''
MSE loss function
Calculate the square of the difference between the output yand the true labeltarget
torch.nn.MSELoss(size_average=None, reduce=None, reduction='mean')
Binary classification loss function
Calculate the logistic loss of the two classifications
torch.nn.SoftMarginLoss(size_average=None, reduce=None, reduction='mean')
Cross entropy loss function
loss_fc = nn.CrossEntropyLoss()
4.1.4 Optimization method:
We can define related methods by ourselves through class definition , or simply by calling functions. Here I only introduce the common methods later: (If other cases are involved, I will add them)
# 1.实例化优化器
# 参数： 参数、学习率、动量大小
optimizer = torch.optim.SGD([weight], lr=0.1, momentum=0.9)
# 2. 梯度更新、参数更新
optimizer.step()
# 3. 输出参数
optimizer.param_groups
# 4. 添加参数组
optimizer.add_param_group({
"params": weight2, 'lr': 0.0001, 'nesterov': True})
Similarly, if we know the name of the optimizer, we can use it, because pytorch is defined:
torch.optim.Adagrad
torch.optim.Adam
torch.optim.SGD
torch.optim.RMSprop
# 常用的优化方法
In addition , some common properties and methods of optimization objects are briefly introduced below:
param_groups: The managed parameter group is a list, each element of which is a dictionarystep(): Perform one-step gradient update, parameter updateadd_param_group(): add parameter group
4.1.5 Data flattening:
Suppose your data is 1-dimensional data, then the data is naturally flattened. If it is 2-dimensional data, then flattening is to change the 2-dimensional data into 1-dimensional data. If it is 3-dimensional data, then it must be based on You choose the "flattening degree" to operate, assuming that all flattening is required, then directly change the 3D data into 1D data, if only partial flattening is required.
method
torch.flatten(obj,start,end)
'''
参数:
obj：操作的对象
start：开始flatten的维度，可以不指定，默认为0
end：结束flatten的维度,可以不指定，不指定默认为-1
返回值：
新的数据
'''
Understand carefully:
# part 1
a = torch.arange(0,5,1) # tensor([0, 1, 2, 3, 4])
print(a.size()) # torch.Size([5])
b = torch.flatten(a)
print(b) # tensor([0, 1, 2, 3, 4])
# part2
a = torch.ones((3,3,3))
print(a.size()) # torch.Size([3, 3, 3])
b = torch.flatten(a)
print(b.size()) # torch.Size([27])
It is not difficult to see from writing in this way directly becomes 1-dimensional data.b = torch.flatten(a)
Look at the following code:
# part3
a = torch.ones((3,3,3))
print(a.size()) # torch.Size([3, 3, 3])
b = torch.flatten(a,start_dim=0)
print(b.size()) # torch.Size([27])
c = torch.flatten(a,start_dim=1)
print(c.size()) # torch.Size([3, 9])
d = torch.flatten(a,start_dim=2)
print(d.size()) # torch.Size([3, 3, 3])
e = torch.flatten(a,start_dim=3)
print(e.size()) # 报错
f = torch.flatten(a,start_dim=1,end_dim=2)
print(f.size()) # torch.Size([3, 9])
From the above code, it can be seen that start_dim and end_dim are counted from 0, the so-called start_dim is the dimension from which to stretch, and end_dim is the dimension from which to end. For example, in the above c = torch.flatten(a,start_dim=1), stretching starts from the second dimension, then the first dimension remains unchanged, and the result is 3*9 (9 is the result of the following 3*3).
4.1.6 Model parameter supplement:
Model parameters are divided into two parts: **all parameters of the model as a whole, and parameters of each layer. ** There are different methods for different objects:
- The overall parameters of the model:
obj = model.parameters()
# 返回一个可迭代的参数对象
- The parameters of each layer
# 一般用于模型构建类中，用法都是
self.layer_obj.weight # 除去偏置外的参数
self.layer_obj.bias # 偏置
4.2 Definition of each layer of the model:
Below, introduce the definitions of various layers of pytorch, which are mainly used to build neural network models.
4.2.1 Linear layer/fully connected layer:
torch.nn.Linear
The main three parameters are:
1. 输入特征数
2. 输出特征数
3. 是否启用偏置
**It is worth noting that: **The weight parameters and offsets of the corresponding dimensions will be automatically generated. For the generated weight parameters and offsets, our model uses a better parameter initialization method than the previous simple random method by default. .
4.2.2 Convolution layer:
torch.nn.Conv2d
# Conv1d一般用于nlp，根据我猜测应该是指的是一维卷积核
The main parameters are:
1. 输入通道数
2. 输出通道数
3. 卷积核大小
4. 卷积核移动步长和paddingde值
4.2.3 Max pooling layer
torch.nn.MaxPool2d
The main parameters are:
1. 池化窗口的大小
2. 池化窗口移动步长和paddingde值
4.2.4 Average pooling layer:
average_pool = nn.AvgPool2d(2,stride=2)
# 参数也是kernel_size和步长
4.2.5 dropout layer:
torch.nn.Dropout(p) # 参数p指定丢失概率
4.2.6 BN layer:
torch.nn.BatchNorm2d(num_features, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
# affine定义了BN层的参数γ和β是否是可学习的(不可学习默认是常数1和0).
4.3 The storage method of the model:
Model storage refers to putting each layer into a container, and this different container is our different model storage method.
4.3.1 Sequential：
module
torch.nn.Sequential
effect
can be used to define ordered layers, and the forward calculation of the model is calculated according to the order.
method
There are two ways to implement it:
- Direct layer-by-layer definition
import torch.nn as nn
net = nn.Sequential(
nn.Linear(784, 256),
nn.ReLU(),
nn.Linear(256, 10),
)
print(net)
- join via an ordered dictionary
import collections
import torch.nn as nn
net2 = nn.Sequential(collections.OrderedDict([
('fc1', nn.Linear(784, 256)),
('relu1', nn.ReLU()),
('fc2', nn.Linear(256, 10))
]))
print(net2)
Advantages and disadvantages
The advantage of using Sequentialthe defined model is that it is simple and easy to read. At the same time, Sequentialthe defined model does not need to be written again forward, because the order has already been defined (this will be discussed later) .
But using it Sequentialwill also make the model definition lose flexibility. For example, when an external input needs to be added in the middle of the model, it is not suitable for Sequentialimplementation.
4.3.2 ModuleList：
module
torch.nn.ModuleList
effect
ReceivesModuleList a list of submodules (or layers, which must belong to nn.Modulea class) as input, and can then Listappend and extend similarly. At the same time, the weights of sub-modules or layers are automatically added to the network.
method
First, define a ModuleList:
net = nn.ModuleList([nn.Linear(784, 256), nn.ReLU()])
net.append(nn.Linear(256, 10)) # # 类似List的append操作
print(net[-1]) # 类似List的索引访问
print(net)
Then , the forward method needs to be defined to form a network:
class model(nn.Module):
def __init__(self, ...):
super().__init__()
self.modulelist = ...
def forward(self, x):
# 指定计算的顺序即可
for layer in self.modulelist:
x = layer(x)
return x
This shows that the calculation method is 输出数据 = 层(输入数据)(described later)
4.3.3 ModuleDict：
module
torch.nn.ModuleDict
effect
ModuleDictSimilar to ModuleList, except that ModuleDictit is more convenient to add names to the layers of the neural network.
method
First of all, define ModuleDict first, otherwise what is received at this time is a dictionary:
net = nn.ModuleDict({
'linear': nn.Linear(784, 256),
'act': nn.ReLU(),
})
net['output'] = nn.Linear(256, 10) # 添加
print(net['linear']) # 访问
print(net.output) # 访问
print(net)
Then, it is the definition order, which is the same as the ModuleList above.
4.4 Model definition:
The above methods, such as Sequnetial\ModuleListetc., are actually containers for the model layer, not our model objects.
To build a model object, you need to customize the model class, but this class needs to be inherited from torch.nn.Module. In addition, this class also has some methods that must be defined and commonly used methods:
import torch
from torch import nn
class MLP(nn.Module):
# 声明带有模型参数的层
def __init__(self):
# 调用MLP父类Block的构造函数来进行必要的初始化。这样在构造实例时还可以指定其他函数
super(MLP, self).__init__()
# 然后，我们需要在这里把我们的神经网络的各个层定义出来，可以使用sequential来定义
self.model = nn.Sequnetial(xxxxxx)
# 定义模型的前向计算，即如何根据输入x计算返回所需要的模型输出
# 即，我们必须把我们的模型存储对象中的各个层定义一个顺序
def forward(self, x):
pass
**The above two methods must be defined. **In addition, the definition order of the forward algorithm is different for different storage methods:
# 比如Sequential，本身就是按照顺序装入各个层的，因此不需要定义顺序，直接：
class MLP(nn.Module):
def __init__(self):
super(MLP, self).__init__()
self.model = nn.Sequnetial(xxxxxx)
def forward(self, x):
self.model(x) # Sequential
# 而对于ModuleLIst/ModuleDIct方法，就必须为它定义顺序，你可以使用：
class MLP(nn.Module):
def __init__(self):
super(MLP, self).__init__()
self.model = nn.ModuleDict/ModuleList(xxxxxx)
def forward(self, x):
# 方式1
for layer in self.model:
xxx
# 方式2
x = self.model[1](x)
x = self.model[3](x)
x = self.model[2](x)
.....
In addition, there are some very common methods:
self.modules()
# 返回一个迭代对象，里面的内容为当前模型的所有层
'''
ex:
for m in self.modules():
pass
'''
self.train(mode=True)
# 设置当前模式为训练模式
self.eval(mode=True)
# 设置当前模式为验证模式
'''
上面两个方法主要是针对一些算法而设置的，因为一些算法在训练与测试上表现不一致，因此才需要设置当前模型正在干什么
'''
5. Data loading and reading:
PyTorch data reading is completed through Dataset+DataLoader. Dataset defines the data format and data transformation form, and DataLoader continuously reads in batches of data in an iterative manner.
Here , we also need to be clear: due to the huge amount of data in deep learning, it is generally not directly loaded all the data, but loaded and trained in batches.
We can create our own data loading function by inheriting the DataSet class, or we can directly call the corresponding method to achieve it.
5.1 Folder organization:
The general image data can be obtained in the following format:
大文件夹
训练集
测试集
train.txt
test.txt
Among them, the general training set and test set will not give you a named file, but directly store the file in order, and the txt file is the corresponding relationship between the stored picture and the label.
5.2 Dataset production:
The general processing flow of DataLoader is:
原始数据 --- 清洗打乱顺序 --- 使之变为可迭代对象 --- 使之变为batch
For the file format we mentioned above , we need to do the following preprocessing:
# 1. 处理train.txt中的“图片名字 标签”，将两者进行一定的存储
# 2. 将名字、标签分开存储于两个列表中
# 3. 将图片名字与绝对路径拼接成为一个真实可靠的路径列表
Next, start making the Dataset class:
# 1. 导入库
from torch.utils.data import Dataset,DataLoader
# 2. 定义类
class MyDataset(Dataset):
def __init__(self,root_dir,ann_file,transform=None):
self.ann_file = ann_file # 图片名字
self.root_dir = root_dir # 绝对路径（不含文件名字）
self.img_lable = self.load_annotations() # 图像名字和标签字典，key为图片名字，value为标签值
self.img = [os.path.join(self.root_dir,img) for img in list(self.img_label.keys())]
self.img = [label for label in list(self.img_label.values())]
self.transform = transform # 对数据的预处理，有则写，无则不写
def __len__(self):
# 返回数据的样本个数，这是调用len(xxx)返回的结果（个人认为）
return len(self.img)
def __getitem__(self,idx):
# idx为固定的参数，每次调用的时候会随机参数一个idx来进行索引
# 必须定义的函数，表示你迭代时获取的值
image = Image.open(self.img[idx]) # 打开图片
label = self.label[idx] # 获取标签值
if self.transform:
# 如果有预处理+转为tensor格式
image = self.transform(image)
# 转为tensor数据
label = torch.from_numpy(np.array(label))
return image,label
def load_annotations(self):
# 自定义的函数，主要是处理train,txt文件
# 返回的是我们上面所说预处理第一步
data_infos = {
}
with open(self.ann_file) as f:
samples = [x.strip().split(' ') for x in f.readlines()]
for filename,gt_label in samples:
data_infos[filename] = np.array(gt_label,dtype=np.int64)
return data_infos
5.3 DataLoader uses:
After we have defined the Dataset, we can use DataLoader to load our data.
You can test it before using it:
# 1. 实例化Dataset
train_dataset = MyDataset(...)
# 2. 实例化DataLoader
train_loader = DataLoader(train_dataset,batch_szie=64,shuffle=True)
'''
必须指定参数：
batch_size: 每批的数据个数
shuffle: 是否打乱数据
'''
# 3. 小迭代试试
image,label = iter(train_loader).next()
If you really train, roughly write like this:
for epoch in range(num):
# 表示训练多少次
for image,label in train_loader:
# 开始训练
xxxxxx
Detailed parameter introduction:
* num_workers：（数据类型 Int）
使用多少个子进程来导入数据
* drop_last：（数据类型 bool）
丢弃最后数据
5.4 Other methods—ImageFloder:
ImageFolder assumes that all files are saved in folders, and images of the same category are stored in each folder, and the folder name is the class name , so that this method can be used to quickly read and load data.
method
dataset = ImageFolder()
'''
主要参数：
root：在root指定的路径下寻找图片
返回值：
dataset对象，但是这个对象有几个常用方法：
dataset.classes --- 根据文件夹名字返回值
dataset.class_to_idx --- 按照顺序将类别定义为索引
dataset.imgs --- 图片路径和标签，返回一个列表，里面为元组
'''
example
from torchvision.dataset import ImageFloder
dataset = ImageFloder('./xxx/')
dataloader = DataLoader(dataset,batch_size=64,shuffle=True)
for image,label in dataloader:
xxxx
5.5 Use of self-contained data:
Load the MNIST dataset
import torchvision
import torch
import torchvision.transforms as transforms
batch_size = 256
train_dataset = torchvision.datasets.MNIST(root='../../data',
train=True,
transform=transforms.ToTensor(),
download=True)
test_dataset = torchvision.datasets.MNIST(root='../../data',
train=False,
transform=transforms.ToTensor())
# Data loader
train_loader = torch.utils.data.DataLoader(dataset=train_dataset,
batch_size=batch_size,
shuffle=True)
test_loader = torch.utils.data.DataLoader(dataset=test_dataset,
batch_size=batch_size,
shuffle=False)
CIFAR-10
from torchvision import datasets
cifar10 = datasets.CIFAR10(root="C:\\cifar10_dataset", train=True, download=True)
'''
参数：
root: cifar10数据集存放目录
train: True，表示加载训练数据集，False，表示加载验证数据集
download: True，表示cifar10数据集在root指定的文件夹不存在时，会自动下载
'''
6. Model training and evaluation:
After you have completed the definition, construction, parameter initialization, loss function setting and other operations of the model, you can train the model.
The model training method is very simple, the general routine is:
# 1. 设置模型状态
model.train()
# 2. 数据加载、梯度训练、模型训练、计算损失值、参数更新
train_loss = 0
for data, label in train_loader:
# 将数据放到GPU上
data, label = data.cuda(), label.cuda()
# 梯度清零
optimizer.zero_grad()
# 将数据送入模型中
output = model(data)
# 计算损失函数值
loss = criterion(label, output)
# 反向传播，更新参数
loss.backward()
optimizer.step()
train_loss += loss.item()*data.size(0)
train_loss = train_loss/len(train_loader.dataset)
7. Visualization:
7.1 torchinfo：
torchinfo visualizes the network structure
First, you need to follow the torchinfo module
pip install torchinfo
Second, you can use it to view the network structure:
torchinfo.summary(model.input_size)
# model : 模型对象
# input_size : 输入大小
7.2 tensorboard：
tensorboard visual training
First, you need to install the tensorboard tool (you can also use the tensorboard that comes with pytorch):
pip install tensorboardX
We can regard TensorBoard as a recorder, which can record the data we specify, including the feature map, weight, and training loss of each layer of the model. TensorBoard saves the recorded content in a user-specified folder, and TensorBoard will keep recording when the program is running continuously. The recorded content can be visualized in the form of web pages.
First , we need to configure it:
# 导入模块
from tensorboardX import SummaryWriter
# 指定保存的文件夹
writer = SummaryWriter(路径)
Then , use it:
- When we have created a model, we can display its model structure:
writer.add_graph(model, input_to_model = torch.rand(1, 3, 224, 224))
# input_to_model : 输入大小
writer.close()
- It is also possible to display graphs for continuous variables
writer = SummaryWriter('./pytorch_tb')
for i in range(500):
x = i
y = x**2
writer.add_scalar("x", x, i) #日志中记录x在第step i 的值
writer.add_scalar("y", y, i) #日志中记录y在第step i 的值
writer.close()
# 上面的效果是两张图显示
Finally , start it:
tensorboard --logdir=/path/to/logs/ --port=xxxx
# /path/to/logs 中 path是保存的文件夹路径，需要修改，其它不用改
# --port : 是服务器使用的时候修改
7.3 Examples:
Experience in practice:
- Generally, visual variables are all visual loss values, but note that the index of the loss value (that is, the third parameter) cannot be repeated. Once repeated, it will be repeated
- Generally do not specify the path, the default is to create a folder named runs under the current folder
- To visualize the network architecture, you need to give him an input, usually batch_data, see the code below
- To see the visualization, you need to call the cmd command
- For specific visualization results, see the figure below
# 导包
import time
import torch
from torch import nn
from torch import optim
from torch.utils.data import DataLoader
from torchvision.datasets import MNIST
import torchvision.transforms as transforms
# 导入可视化
from tensorboardX import SummaryWriter
# 创建模型
class LeNet(nn.Module):
def __init__(self):
super(LeNet,self).__init__()
# 定义模型
self.features = nn.Sequential(
nn.Conv2d(in_channels=1,out_channels=6,kernel_size=(5,5),stride=1,padding=2),
nn.Sigmoid(),
nn.MaxPool2d(kernel_size=2,stride=2),
nn.Conv2d(in_channels=6,out_channels=16,kernel_size=(5,5),stride=1),
nn.Sigmoid(),
nn.MaxPool2d(kernel_size=2,stride=2),
)
self.classifier = nn.Sequential(
nn.Linear(in_features=400, out_features=120),
nn.Sigmoid(),
nn.Linear(in_features=120, out_features=84),
nn.Sigmoid(),
nn.Linear(in_features=84, out_features=10)
)
def forward(self,x):
# 定义前向算法
x = self.features(x)
# print(x.shape)
x = torch.flatten(x,1)
# print(x.shape)
result = self.classifier(x)
return result
# 下载数据集或者加载数据集
train_dataset = MNIST(root='../data',train=True,transform=transforms.ToTensor(),download=True)
test_dataset = MNIST(root='../data',train=False,transform=transforms.ToTensor())
# 加载数据: 分批次，每批256个数据
batch_size = 32
train_loader = DataLoader(dataset=train_dataset,batch_size=batch_size,shuffle=True)
test_loader = DataLoader(dataset=test_dataset,batch_size=batch_size,shuffle=False)
# start time
start_time = time.time()
# 创建模型
model = LeNet()
# 模型放入GPU中
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
model = model.to(device)
# 定义损失函数
loss_func = nn.CrossEntropyLoss()
loss_list = [] # 用来存储损失值
# 定义优化器
SGD = optim.Adam(params=model.parameters(),lr=0.001)
# 初始化可视化对象
writer = SummaryWriter() # 路径一般默认，不过也可以指定
# 训练指定次数
x = 0
for i in range(10):
loss_temp = 0 # 定义一个损失值，用来打印查看
# 其中j是迭代次数，data和label都是批量的，每批32个
for j,(batch_data,batch_label) in enumerate(train_loader):
# 启用GPU
batch_data,batch_label = batch_data.cuda(),batch_label.cuda()
# 清空梯度
SGD.zero_grad()
# 模型训练
prediction = model(batch_data)
# 计算损失
loss = loss_func(prediction,batch_label)
loss_temp += loss
# BP算法
loss.backward()
# 更新梯度
SGD.step()
if (j + 1) % 200 == 0:
print('第%d次训练，第%d批次，损失值: %.3f' % (i + 1, j + 1, loss_temp / 200))
# 可视化1：一般添加loss值
writer.add_scalar('200_step_loss',loss_temp / 200,x)
x += 1
loss_temp = 0
# end_time
end_time = time.time()
print('训练花了: %d s' % int((end_time-start_time)))
# 可视化2：模型结构
writer.add_graph(model,input_to_model=batch_data)
# 关闭可视化
writer.close()
Let's see the result:
8. Call the GPU:
8.1 Check if GPU can be used:
Detection code:
import torch
# 检测是否可以使用GPU
print(torch.cuda.is_available())
8.2 View the number and name of GPUs:
Code:
# 查看GPU数量，索引号从0开始
print(torch.cuda.current_device())
# 根据索引号查看GPU名字
print(torch.cuda.get_device_name(0))
8.3 Calling steps:
The core steps are as follows:
# 1.创建模型
model = Net()
# 2.定义device
device = torch.device("cuda:0" if torch.cuda.is_available() else "cpu")
# 3.将模型加载到GPU(所定义的device)
model = model.to(device)
# 4.将输入和输出加载到GPU
inputs, target = inputs.to(device), target.to(device)
Let's talk about how to use it in detail:
- The difference between cuda:0 and cuda
device = torch.device('cuda')
device = torch.device('cuda:0')
'''
对于单卡计算机都一样，但是对于多卡来说，你要使用哪张GPU就需要指定了
当然，你也可以指定CPU，比如：device = torch.device('cpu')
'''
toA few notes about the method
# 1. 想要将一个张量放到GPU上，可以使用to方法，也可以使用cuda，看下面代码：
import torch
# 创建一个device
device = torch.device('cuda:0')
# 创建一个tensor
a = torch.randn((3,4)).to(device)
# 等价于 a = a.cuda(),同样返回新值
# 等价于torch.randn((3,4),device=device)
print(a.device) # cuda:0
# 注意： to()方法返回一个新值，而不是修改原来的a
**About how to use: **Basically put the model into the GPU first, then put the data into the GPU, and the rest of the operations are the same.
For example, pseudocode:
# 创建模型
model = MyModel()
# 模型放入GPU
model = model.to(deivce)
# 训练
for image,label in train_loader:
image,label = image.to(device),label.to(device)
# 正常训练
9. torchvision module:
9.1 Transforms commonly used methods:
Commonly used methods:
Additionally, there is a sequentialmethod similar to:
transforms.Compose([
# transforms.xxx
])
9.2 Common methods of models:
The models library in torchvision is a library that contains common classic networks, such as AlexNet, VGG, ResNet, etc. that we are familiar with.
The calling method is very simple, first know the network name, and then call it according to the following format:
torchvision.models.alexnet
However, the relevant parameters still need to check the documentation by yourself.
In addition , according to viewing the source code, you can know that calling this method is actually similar to implementing it yourself.
10. Model saving and loading:
10.1 Saving and loading the entire model:
# 保存和加载整个模型
torch.save(model_object, 'model.pkl')
model = torch.load('model.pkl')
10.2 Save and load model parameters:
# 仅保存和加载模型参数(推荐使用)
torch.save(model_object.state_dict(), 'params.pkl')
model_object.load_state_dict(torch.load('params.pkl'))
Talk about how to use:
- The difference between cuda:0 and cuda
device = torch.device('cuda')
device = torch.device('cuda:0')
'''
对于单卡计算机都一样，但是对于多卡来说，你要使用哪张GPU就需要指定了
当然，你也可以指定CPU，比如：device = torch.device('cpu')
'''
toA few notes about the method
# 1. 想要将一个张量放到GPU上，可以使用to方法，也可以使用cuda，看下面代码：
import torch
# 创建一个device
device = torch.device('cuda:0')
# 创建一个tensor
a = torch.randn((3,4)).to(device)
# 等价于 a = a.cuda(),同样返回新值
# 等价于torch.randn((3,4),device=device)
print(a.device) # cuda:0
# 注意： to()方法返回一个新值，而不是修改原来的a
**About how to use: **Basically put the model into the GPU first, then put the data into the GPU, and the rest of the operations are the same.
For example, pseudocode:
# 创建模型
model = MyModel()
# 模型放入GPU
model = model.to(deivce)
# 训练
for image,label in train_loader:
image,label = image.to(device),label.to(device)
# 正常训练
9. torchvision module:
9.1 Transforms commonly used methods:
Commonly used methods:
[External link image transfer...(img-CpvdHnXg-1690970949373)]
Additionally, there is a sequentialmethod similar to:
transforms.Compose([
# transforms.xxx
])
9.2 Common methods of models:
The models library in torchvision is a library that contains common classic networks, such as AlexNet, VGG, ResNet, etc. that we are familiar with.
The calling method is very simple, first know the network name, and then call it according to the following format:
torchvision.models.alexnet
However, the relevant parameters still need to check the documentation by yourself.
In addition , according to viewing the source code, you can know that calling this method is actually similar to implementing it yourself.
10. Model saving and loading:
10.1 Saving and loading the entire model:
# 保存和加载整个模型
torch.save(model_object, 'model.pkl')
model = torch.load('model.pkl')
10.2 Save and load model parameters:
# 仅保存和加载模型参数(推荐使用)
torch.save(model_object.state_dict(), 'params.pkl')
model_object.load_state_dict(torch.load('params.pkl'))