This article teaches you to use the leased GPU platform to run yolov5

written in front

This article is a record of the author's learning process a few days ago. Since there are few such articles, it is convenient for latecomers to write it.
This article focuses on the rapid use of yolov5, and the principle part is less generalized. I hope you can successfully perform target detection after reading this article.
GPU rental platform: https://www.autodl.com/home
yolov5 official code: https://github.com/ultralytics/yolov5
author changed code: link: https://pan.baidu.com/s/1oZWhHjAy_Wp4mg6doaSrkQ Extraction code: f7re
data set: link: https://pan.baidu.com/s/1Ra7bf5JQavIA69kNLiUnXA Extraction code: 6z0j
filezilla download address: client-FileZilla Chinese website

Part of the previous article involves the configuration of yolov5, please jump directly to the third part for renting GPU.

text

1. Download the yolov5 code file

For the environment configuration of yolov5, you can check other blogs or a treasure for help, so I won’t go into details here, the main thing is to install pycharm professional version and pytorch environment.
After the environment is ready, you can download yolov5 (the second link at the beginning of the article), and you can get a compressed package by following the steps below.
insert image description here
Unzip the compressed package into the code path you prepared, and open pycharm. Note that this article uses the leased GPU platform to run yolov5, and only the professional version of pycharm can be used. If it is not the professional version, you can ask other posts or a certain treasure for help. To distinguish whether it is a professional version, you can check it on the software startup page, and the word (Professional) is a professional version.
insert image description here
When you come to the initial interface, select the compressed package you just decompressed, select the folder and open it. After opening, you will see this interface. Just opened it is a readme file.

Second, make changes to the yolov5 file

Using yolov5 to train the target detection model, the main changes involved are some parameters and paths of the data set and training files. The configuration of the dataset is first given below, and the tutorial is applicable to both public and private datasets.

1. Establish a data set path

Open the yolov5-master file, create a detection (can be different, but English) folder in the same directory as train.py, then create a dataset folder in the detection folder, and continue to create images and labels in the dataset folder. Finally, create the train and test folders in these two folders to store the training set and test set respectively. The folder directory and structure are as follows:
insert image description here

2. Add dataset file

Copy the coco128.yaml file in the yolov5-master/data folder to the yolov5-master/detection folder, and change the folder name to detection.yaml. The coco128.yaml here is a small data set consisting of the first 128 images in COCO train2017. The settings contained in the file include the image list path, the training set verification set path, and the number and name of classes.

We need to make changes according to the dataset path and categories in the dataset we just created. The changes are as follows:

insert image description here

Then copy the yolov5x.yaml file in the yolov5-master/models folder to the yolov5-master/detection folder. Of course, s, n, m, and l models can also be used. Here we only take the yolov5x.yaml model as an example. , this file mainly includes parameter configuration, anchors configuration, and network structure. We only need to change the nc value to the number of dataset categories.

insert image description here

The data set we chose is a cat and dog data set, and the data set link can be downloaded at the beginning of the article. Now we have 500 pictures of cats and dogs in our hands, and we need to label the pictures first. For the use of annotation software, you can check other blogs. The data set link provided at the beginning of the article has been annotated and can be used directly.

After downloading the data set, since yolov5 only recognizes txt files but not xml, it is necessary to complete the conversion from xml to txt files. The conversion procedure is as follows:

import xml.etree.ElementTree as ET
import pickle
import os
from os import listdir, getcwd
from os.path import join


def convert(size, box):
    # size=(width, height)  b=(xmin, xmax, ymin, ymax)
    # x_center = (xmax+xmin)/2        y_center = (ymax+ymin)/2
    # x = x_center / width            y = y_center / height
    # w = (xmax-xmin) / width         h = (ymax-ymin) / height

    x_center = (box[0] + box[1]) / 2.0
    y_center = (box[2] + box[3]) / 2.0
    x = x_center / size[0]
    y = y_center / size[1]

    w = (box[1] - box[0]) / size[0]
    h = (box[3] - box[2]) / size[1]

    # print(x, y, w, h)
    return (x, y, w, h)


def convert_annotation(xml_files_path, save_txt_files_path, classes):
    xml_files = os.listdir(xml_files_path)
    # print(xml_files)
    for xml_name in xml_files:
        # print(xml_name)
        xml_file = os.path.join(xml_files_path, xml_name)
        out_txt_path = os.path.join(save_txt_files_path, xml_name.split('.')[0]+'.txt')
        out_txt_f = open(out_txt_path, 'w')
        tree = ET.parse(xml_file)
        root = tree.getroot()
        size = root.find('size')
        w = int(size.find('width').text)
        h = int(size.find('height').text)

        for obj in root.iter('item'):
            # difficult = obj.find('difficult').text
            cls = obj.find('name').text
            # if cls not in classes or int(difficult) == 1:
            #     continue
            cls_id = classes.index(cls)
            xmlbox = obj.find('bndbox')
            b = (float(xmlbox.find('xmin').text), float(xmlbox.find('xmax').text), float(xmlbox.find('ymin').text),
                 float(xmlbox.find('ymax').text))
            # b=(xmin, xmax, ymin, ymax)
            # print(w, h, b)
            bb = convert((w, h), b)
            out_txt_f.write(str(cls_id) + " " + " ".join([str(a) for a in bb]) + '\n')


if __name__ == "__main__":
    # 把forklift_pallet的voc的xml标签文件转化为yolo的txt标签文件
    # 1、需要转化的类别
    classes = ['cat', 'dog']
    # 2、voc格式的xml标签文件路径
    xml_files1 = r'***'
    # 3、转化为yolo格式的txt标签文件存储路径
    save_txt_files1 = r'***'
    convert_annotation(xml_files1, save_txt_files1, classes)

After converting to a txt file, the data set can be divided manually or using a program. In view of the small size of the data set this time, I chose to manually divide the training set and the test set at a ratio of 4:1, that is, randomly select 50 cat and dog photos each and put them in the test folder, and put the remaining 400 pictures in the train folder , note that the label also operates in the same way, and at the same time ensure that the label name corresponds to the image name.

3. Change training parameters

The training parameters are mainly to change the train.py and detect.py parts, where the changes in train.py are used to train the model, and the changes in detect.py are used to detect the training results.

Open the train.py file and go to line 435. Some parameter descriptions and changes are as follows:

Line 437, weights part, we use yolov5x.pt this time, so change ROOT / 'yolov5s.pt' after default to ROOT / 'yolov5x.pt'

Line 438, configure the network structure part, corresponding to yolov5x.yaml, so fill in ROOT / 'detection/yolov5x.yaml' after default

Line 439, the data set configuration file, corresponds to detection.yaml, so fill in ROOT / 'detection/detection.yaml' after default

Line 441, training rounds, determine the training rounds according to the training results of the dataset

Line 454, training equipment, generally uses GPU0, so fill in 0 after default

Other parameters do not need to be changed if it is the first training for Xiaobai, and can be changed as needed after getting familiar with it.

The changed parameter code is as follows:

     parser.add_argument('--weights', type=str, default=ROOT / 'yolov5x.pt', help='initial weights path') #weight权重路径，需更改
    parser.add_argument('--cfg', type=str, default=ROOT / 'detection/yolov5x.yaml', help='model.yaml path') #cfg配置文件（网络结构），需更改
    parser.add_argument('--data', type=str, default=ROOT / 'detection/detection.yaml', help='dataset.yaml path') #数据集配置文件（路径），需更改
    parser.add_argument('--hyp', type=str, default=ROOT / 'data/hyps/hyp.scratch-low.yaml', help='hyperparameters path') #hyp超参数文件，根据需要更改
    parser.add_argument('--epochs', type=int, default=100, help='total training epochs') #训练轮次，根据需要更改
    parser.add_argument('--batch-size', type=int, default=2, help='total batch size for all GPUs, -1 for autobatch') #训练批次，根据需要更改，配置较低时设置小一些
    parser.add_argument('--imgsz', '--img', '--img-size', type=int, default=640, help='train, val image size (pixels)') #设置图片大小，根据需要更改
    parser.add_argument('--rect', action='store_true', help='rectangular training')
    parser.add_argument('--resume', nargs='?', const=True, default=False, help='resume most recent training') #是否接着上次训练结果，继续训练
    parser.add_argument('--nosave', action='store_true', help='only save final checkpoint')
    parser.add_argument('--noval', action='store_true', help='only validate final epoch') #最后进行测试
    parser.add_argument('--noautoanchor', action='store_true', help='disable AutoAnchor')
    parser.add_argument('--noplots', action='store_true', help='save no plot files')
    parser.add_argument('--evolve', type=int, nargs='?', const=300, help='evolve hyperparameters for x generations')
    parser.add_argument('--bucket', type=str, default='', help='gsutil bucket')
    parser.add_argument('--cache', type=str, nargs='?', const='ram', help='image --cache ram/disk') #是否提前缓存图片到内存，以加快训练速度
    parser.add_argument('--image-weights', action='store_true', help='use weighted image selection for training')
    parser.add_argument('--device', default='0', help='cuda device, i.e. 0 or 0,1,2,3 or cpu') #选择训练设备，一般用GPU0
    parser.add_argument('--multi-scale', action='store_true', help='vary img-size +/- 50%%')
    parser.add_argument('--single-cls', action='store_true', help='train multi-class data as single-class')
    parser.add_argument('--optimizer', type=str, choices=['SGD', 'Adam', 'AdamW'], default='SGD', help='optimizer')
    parser.add_argument('--sync-bn', action='store_true', help='use SyncBatchNorm, only available in DDP mode')
    parser.add_argument('--workers', type=int, default=8, help='max dataloader workers (per RANK in DDP mode)')
    parser.add_argument('--project', default=ROOT / 'runs/train', help='save to project/name') #保存路径
    parser.add_argument('--name', default='exp', help='save to project/name')
    parser.add_argument('--exist-ok', action='store_true', help='existing project/name ok, do not increment')
    parser.add_argument('--quad', action='store_true', help='quad dataloader')
    parser.add_argument('--cos-lr', action='store_true', help='cosine LR scheduler')
    parser.add_argument('--label-smoothing', type=float, default=0.0, help='Label smoothing epsilon')
    parser.add_argument('--patience', type=int, default=100, help='EarlyStopping patience (epochs without improvement)') #早停止忍耐次数，100次不更新就停止训练

Note: The detect part is recommended to watch after the training model is completed

Open the detect.py file and go to line 221. After the training is completed, we only need to change the three parts of weights source data to recognize, and other parameters can be changed as needed.
insert image description here

Line 221, the weights part, for the trained weight file, such as 'runs/train/exp4/weights/best.pt'

Line 222, the picture to be detected, fill in the path by yourself

Line 223, the data set configuration file, corresponds to detection.yaml, so fill in ROOT / 'detection/detection.yaml' after default

3. Lease GPU

Open the Wenshou GPU rental platform and register an account. If you are a student, you can apply for student certification, so that you can get a 5% discount on GPU rental.

Go to the main interface, select an available GPU, and rent it. For example, click "One card can be rented" in the picture below

insert image description here

Then the billing method, you can choose pay-as-you-go, daily, weekly or monthly. I usually use pay-as-you-go, which is more economical.

insert image description here

Then in the mirroring part, you can choose the algorithm mirroring, and then search for yolov5 in it. I understand that the algorithm mirroring is to provide the environment required for the algorithm.

insert image description here

After all these settings are completed, it is enough to rent.

Note: For operations such as uploading resources and changing codes, you can use the no-card mode to boot, and the billing will be much less, and the GPU will be used when training the model.

4. Use the leased GPU to run yolov5

1. Use of filezilla

After downloading the filezilla software at the beginning of the article, install it and come to the main interface:
insert image description here

Then select File->Site Manager in the upper left corner, enter, create a new site first, and then fill in other information according to the server information you leased, and fill in SFTP in the agreement part

insert image description here

Then just click connect, and you can connect to your leased GPU host. If you still don't understand, you can read this official tutorial:

AutoDL Help Documentation

After the connection, the main interface looks like this. We can upload the local yolov5 folder to the root subfolder (select the local folder and right click to select upload):
insert image description here

2.pycharm connects to AutoDL

Select the interpreter section at the bottom right of pycharm, select Add a new interpreter, then select SSH to add, and follow the steps to enter the user name, password, etc.:

insert image description here

When you reach the last step to create a virtual environment, configure it as shown below:

insert image description here

After completing the above steps, you can successfully use pycharm to connect to the remote GPU. When connecting and initializing, pycharm will update the interpreter, upload resources and other operations. Since we have just uploaded the program using filezilla, we can cancel the process of uploading resources in the process at this time.

Then, we select Tools -> Deployment -> Browse Remote Host to view our remote host on the right side in pycharm.
insert image description here

3. Training model

Select the terminal in pycharm, then select the GPU we rented this time, and enter the command (after python is the absolute path of train.py):

insert image description here

python /root/yolov5-master/train.py

insert image description here

This allows for training. In addition, it should be noted that when modifying the remote file in pycharm, it needs to be uploaded after modification to ensure that it is consistent with the remote file.

insert image description here

The weight file after training is in the runs/train/exp/weights folder, you can choose best.pt or last.pt, and download it through filezilla. If you are not satisfied with the training results this time, you can increase the number of training rounds or change other parameters. Remember to change the weight of the weights part on line 437 in train.py, such as: