Getting started with Pytorch (1) Data loading initialization and training process monitoring

The Pytorch introductory series will generally have less than 5 articles. If you have a chance in the future, you can update it carefully. It mainly reviews the basic knowledge of Pytorch, and reviews the study notes for getting started with Pytorch in the sophomore year! The original tutorial is located at Station B, and I personally feel pretty good about it.
Super Portal , this series of tutorials will quickly let us get started with Pytorch. Although we can't understand the principle of the whole process, we can understand the general process of deep learning training.

---

1. Data Correspondence

• ①The label is marked in the folder name where the data set is located
• ②The label is mixed in the file name of this type of data set in a certain format or marked in the picture
• ③ Store data and labels separately, set a data folder, a label folder, the same file name (remove the suffix) one to store data, one to store labels

Second, the reading of the data set

• ①dataset
  provides a way to obtain the label and data of the dataset
  How to obtain each data and its label
  tells us how much data there is in total
• ②dataloader
  provides different data forms for the subsequent network

You can implement a class that reads data yourself. (Customized magic functions are very similar to generic programming operator overloading in C++)

# torch是pytorch框架的工具箱,utils是工具箱的一个常用的工具包,dataset就是那个工具

from torch.utils.data import Dataset
from PIL import Image
import os
#
class MyData(Dataset):
    # 将数据集所在的文件夹上一级传进去,作为root_dir,数据集所在的文件夹名称作为label_dir
    def __init__(self,root_dir,label_dir):
        self.root_dir=root_dir
        self.label_dir=label_dir
        # 将数据集所在的路径进行完整的拼接存储到path属性中
        self.path=os.path.join(self.root_dir,self.label_dir)
        # 将数据集所在的文件路径传进去,读取出来所有的文件名称
        self.img_path=os.listdir(self.path)
    # 将文件名所处的位置传进去,idx为int型作为数组的下标
    def __getitem__(self, idx):
        # 获取数据的名称
        img_name=self.img_path[idx]
        # 将数据集所在的路径与数据文件名进行拼接
        img_item_path=os.path.join(self.root_dir,self.label_dir,img_name)
        # 从文件夹内读取文件
        img=Image.open(img_item_path)
        # 读取数据的标签
        label=self.label_dir
        # 返回出所读的数据
        return img,label
    def __len__(self):
        return len(self.img_path)

# print(os.getcwd()+"../../数据集/练手数据集/train")
# print(os.path.split(os.getcwd())[0])
# C:\Users\123\Desktop\近期作业\机器学习\9.PyTorch框架(入门)
if __name__=="__main__":
    root_dir=os.path.split(os.getcwd())[0]+r"\数据集\练手数据集\val"
    ants_label_dir='ants'
    bees_label_dir='bees'
    ant_dataset=MyData(root_dir,ants_label_dir)
    bees_dataset=MyData(root_dir,bees_label_dir)
    img1,label1=bees_dataset[0]
    img2,label2=ant_dataset[0]
    trains=ant_dataset+bees_dataset
    img1.show()
    img2.show()
    print(len(ant_dataset))
    print(len(bees_dataset))
    print(type(trains),len(trains))

Three, two magic functions

Pytorch is a toolkit, which contains tools written by others. What should I do if there are too many tools that I can’t use myself?

• dir() : List the functions, or the functions and attributes under the class
• help() : View how to use a function or property

Fourth, editor comparison

• 1.pycharm: An environment for integrated development of python. When running a py file inside it, it will start from the beginning of the file and execute downwards sequentially (this execution will be terminated directly when an error is reported)
• 2. The command line that comes with python: It can be run in modules, but it is difficult to correct errors after running (shift+enter to enter multi-line editing mode)
• 3.jupyter: A multifunctional function interpreter with its own virtual environment, which supports file module operation (it also supports module operation after an error is reported, unless the program crashes) Disadvantages: Each module has dependencies and must be
  run one by one

5. Use of Tensorboard

TensorBoard is a powerful visualization tool and a web application suite that comes with TensorFlow. TensorBoard currently supports 7 visualizations, Scalars, Images, Audio, Graphs, Distributions, Histograms and Embeddings. The main functions of the visualization are as follows.

• (1) Scalars: Displays the changes in accuracy, loss, and weight/bias during training.
• (2) Images: Display the images recorded during the training process.
• (3) Audio: Display the audio recorded during the training process.
• (4) Graphs: Display the data flow graph of the model, as well as the memory and time consumed by training on each device.
• (5) Distributions: Displays the partition diagram of the data recorded during the training process.
• (6) Histograms: Histograms showing the data recorded during training.
• (7) Embeddings: Display the projection division after the word vector.

TensorBoard listens on port 6006 by running a local server. When the browser makes a request, analyze the data recorded during training and draw images during the training process. The visual interface of TensorBoard is shown in the figure below and

the usage method is as follows:

import os
from torch.utils.tensorboard import SummaryWriter
from PIL import Image
import numpy as np
# 指定检测日志存储的位置（可以指定如果不指定的话存进默认的路径）
# Default is runs/**CURRENT_DATETIME_HOSTNAME**,
# 项目基础路径
basepath=os.path.split(os.getcwd())[0]
# 将训练数据记录在项目基础路径下的\logss\logs文件夹内
writer = SummaryWriter(basepath+r'\logss\logs')

# add_scalar()  将数据加入到summary中
# 第一个参数是图表的标题
# 第二个参数是训练的数值（也就是y轴）
# 第三个参数是训练的多少步（也就是x轴）

for i in range(100):
    writer.add_scalar('y=x',i,i)
print("成功将数据导入!")

# add_image()
# 第一个参数是标题,第二个是图片数据(可以是torch.tensor numpy.array string blobname)
# 将图片加入到观测到数据曲线中,用于检测每一步的数据变化,一旦数据有异常
# 可以找出异常数据的准确位置
img=Image.open(os.path.split(os.getcwd())[0]+r"\数据集\hymenoptera_data\train\ants\0013035.jpg")
# print(img)
img.show()
img=np.array(img)
writer.add_image("test",img,1,dataformats="HWC")
print("成功将图片导入!")
writer.close()

The information generated during the training process will be stored in the specified folder, and the naming format is as follows:

We can use the following command to check whether the training process meets current expectations

tensorboard --logdir=logs_train

Usually we will record the following information:

writer.add_scalar("train_loss", loss.item(), total_train_step)
writer.add_scalar("train_acc", train_acc, total_train_step)
writer.add_scalar("test_loss", total_test_loss, total_test_step)
writer.add_scalar("test_acc", test_acc, total_test_step)

6. Transforms data preprocessing

The main function of transforms is to preprocess the data to make the characteristics of the data more obvious. totrnsor converts the data into data that can be passed into the neural network, and adds some attributes or functions inside to facilitate the propagation of pictures in the neural network. It also exists in the object of this class as a totensor attribute.
Commonly used data conversion methods:

• totensor //-------------Add the image to the tensor object

The following three converters must be the parameters of the tensor data type, and the result obtained is also the tensor data type. When generating the converter, it is not necessary to pass in the image, just pass in the parameters used for the conversion. When performing image conversion, the image Pass to the call function of the converter for conversion.

• transforms.Normalize() //-------------Normalization
• transforms.resize() //-------------Modify the image to the specified size
• transforms.randomcrop() //-------------random cropping

# 读取一个图像
basepath=os.path.split(os.getcwd())[0]
img_path=basepath+r"\数据集\hymenoptera_data\train\ants\0013035.jpg"
img=Image.open(img_path)
# 转换为Tensor类型
trans_toten=transforms.ToTensor()
tensorimg=trans_toten(img)
print(tensorimg)
print("---------------正常化前----------------------")
print(tensorimg[0][0][0])
trans_norm=transforms.Normalize([0.1,0.9,0.1],[0.9,0.1,0.9])
trans_img=trans_norm(tensorimg)
print("---------------正常化后----------------------")
print(trans_img[0][0][0])
# 修改大小
# 进去resize之前是PIL图像,出来之后依旧是PIL图像
trans_resize=transforms.Resize((512,512))
resize_img=trans_resize(img)
# 打印可知这两个图像的大小有所差异
print(img)
# img.show()
print(resize_img)

The following converter can perform multiple image conversions. When constructing the converter, pass in the list of converters that need to be converted. The working principle is that the output of the previous converter will be used as the input of the next converter, which is a batch process. The method is similar to the Docker Compose we usually use.
transforms.Compose() performs image transformation step by step

• The parameter is a list, and the parameter passed in is the object generated by the transforms toolkit
• The output result of the previous parameter is used as the input of the next parameter

After generating the converter, you can directly pass in the PIL or numpy array. The method of use is as follows:

# 定义一系列转换器
trans_toten=transforms.ToTensor()
trans_resize=transforms.Resize((512,512))
# 编排
trans_compose=transforms.Compose([trans_resize,trans_toten])
# 转换
compose_img=trans_compose(img)

There may be doubts here:
What is the Tensor data type?
When sending data to the neural network, it is not only necessary to pass in data, but also to pass in parameters. The Tensor data type comprehensively stores commonly used attributes, making the data more suitable. Neural Networks.
How to use Transform
Transform is to convert numpy type data or PIL image into tensor data type
. The principle of use is to generate a tensor object through the transform.ToTensor() class template.

7. Use of Torchvision official data set

It can be seen from the figure below that Torchvision has many pre-processed data sets that come with it, which we can use when testing the model. The following will introduce how to use the CIFAR10 data set.

train_set=torchvision.datasets.CIFAR10(root=basepath+r"\数据集",train=True,download=True,transform=trans_compose)
test_set=torchvision.datasets.CIFAR10(root=basepath+r"\数据集",train=False,download=True,transform=trans_compose)

torchvision.datasets.CIFAR10 parameter explanation

• root specifies the directory where the dataset is stored
• train True represents the training set, False represents the test machine, there are about 5000 training sets and 1000 test sets in the CIFAR10 data set
• download means that if there is no required data set in the specified path, it will be downloaded online, and if there is, it will do nothing
• transform can specify a series of changes of the image can be compose

The code for loading torchvision.datasets.CIFAR10 as a whole:

import os

import torchvision
# 全局取消证书验证
# import ssl
# ssl._create_default_https_context = ssl._create_unverified_context
from torch.utils.tensorboard import SummaryWriter
from torchvision import transforms

basepath=os.path.split(os.getcwd())[0]
# 编排好的数据集处理方式，一会加载数据集的时候使用。
trans_compose=transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize([0.5,0.5,0.5],[0.5,0.5,0.5])
])
train_set=torchvision.datasets.CIFAR10(root=basepath+r"\数据集",train=True,download=True,transform=trans_compose)
test_set=torchvision.datasets.CIFAR10(root=basepath+r"\数据集",train=False,download=True,transform=trans_compose)
# # 打印测试集中的第一个数据（获取到的时一个元组）
# print(test_set[0])
# # 打印测试集中有的数据类型
# print(test_set.classes)
# # 获取测试集第一个数据的数据与标签
# img,target=test_set[0]
# # 打印数据
# print(img)
# # 打印标签
# print(target)
# # 在类别中找到数据对应的类别
# print(test_set.classes[target])
# img.show()
writer=SummaryWriter(basepath+r'\logss\log1')
for i in range(10):
    # 因为test_set存放的是所有图片信息以及数据集中每一张图片对应的类别还有一些其他配置
    # 使用下标获取到的数据是一个元组,有图片数组有对应的类别
    # 所以对图片操作时先将图片与类别获取出来,然后再将图片传进tensorboard中进行检测
    img,target=test_set[i]
    #使用Tensorboard查看数据集。
    writer.add_image("test",img,i)
writer.close()

Eight, the use of DataLoader

The role of DataLoader
The role of DataSet is to read the data in the form of labels + images.
DataLoader is to extract the data in the DataSet in a certain form and send it to the neural network.
The main parameters of DataLoader

• dataset Data and label mapping relationship storage object
• batch_size The size read each time, that is, how many pictures are packaged and read at one time
• After shuffle is read, whether to rearrange the order of the next read, True to rearrange
• num_workers multithreading, 0 means the main thread
• drop_last Whether the last data is not enough to pack or not (True to discard)

The following is a piece of code that uses DataLoader to load a dataset:

import os
import torchvision
from torch.utils.data import DataLoader
from torch.utils.tensorboard import SummaryWriter
basepath=os.path.split(os.getcwd())[0]
test_data=torchvision.datasets.CIFAR10(root=basepath+r"\数据集",train=False,transform=torchvision.transforms.ToTensor())
# 读取test_data每次读64张图片,读完重新打乱顺序,只使用主线程,删除不够打包的数据
test_loader=DataLoader(dataset=test_data,batch_size=64,shuffle=True,num_workers=0,drop_last=True)
# 探索一下test_loader中都有什么
print(test_loader)
writer=SummaryWriter(basepath+r"\logss\log2")
for test in range(2):
    step=0
    for data in test_loader:
        imgs,targets=data
        writer.add_images(f"test:{
      
      test}",imgs,step)
        step=step+1
        if test==0:
            print(data)
            print(imgs)
            print(targets)
        '''
        会将打包好的图片以及他们对应的标签一块打印出来
        '''
writer.close()