Realizing Rotary Wing UAV Detection: Exploring the Whole Process from Model Training to Actual Deployment

Table of contents

Introduction:

1. Project background

2. Data preprocessing

3. Code implementation

3.1 Install Paddle Detection

3.2 Detection data analysis

3.3 Model Training

3.4 Model Evaluation

3.5 Model Inference

3.6 Model export

3.7 FastDeploy rapid deployment

4. Effect display

---

Introduction:

In this blog post, we detail how to automate the detection of rotary-wing drones, from training to deployment. UAVs have been widely used in various fields, such as aerial photography, logistics, agriculture, etc. However, with the widespread use of UAVs, how to quickly and accurately detect the existence of UAVs is crucial to the safety management and monitoring of UAVs. become particularly important.

In this post, we will start with data collection and labeling, then detail how to train a deep learning model for drone detection, and then how to deploy the trained model to an actual surveillance system. We will comprehensively cover each step of this problem, and provide relevant code and tools, so that readers can follow the steps of the article to implement it by themselves.

1. Project background

This dataset was collected by Mehdi Özel for the drone competition. Most current drone datasets only contain photos taken by drones (mostly drone views of the ground). Unlike other datasets, the images in this dataset are images of drones, which can be used to train our drones to guide and avoid other drones. The dataset has 1359 photos, all with labels. The dataset only includes rotorcraft. Does not include fixed wing. This project trains a target detection model based on this data set, so that the model can detect rotary-wing drones, and achieve the effect of mAP ≥ 0.8 under the self-divided verification set.

2. Data preprocessing

Step01: Decompress the dataset

ERROR1: When I use the unzip command to decompress the data set, the following error occurs.

/bin/bash: -c: 行 0: 未预期的符号 `(' 附近有语法错误
/bin/bash: -c: 行 0: `unzip /home/aistudio/data/data191191/DroneDataset (UAV).zip -d /home/aistudio/work/'

SOLUTION1: Rename the dataset, remove the "()". Namely DroneDataset (UAV).zip -> DroneDataset.zip.

In [ ]

!unzip /home/aistudio/data/data191191/DroneDataset.zip -d /home/aistudio/work/

Step02:  Distinguish files with different suffixes in the folder

This project uses the images and annotation data in dataset_xml_format. Since the images and annotation data are stored together, we first need to store them separately for subsequent processing.

First, we create two new folders /home/aistudio/work/dataset_xml_format in this directory, which are JPEGImages and Annotations respectively.

JPEGImages are used to store images in the dataset.

Annotations are used to store annotation files.

Then move the files with the same suffix to the specified folder through the following command.

In [ ]

!mv /home/aistudio/work/dataset_xml_format/dataset_xml_format/*.png /home/aistudio/work/dataset_xml_format/JPEGImages/
!mv /home/aistudio/work/dataset_xml_format/dataset_xml_format/*.jpg /home/aistudio/work/dataset_xml_format/JPEGImages/
!mv /home/aistudio/work/dataset_xml_format/dataset_xml_format/*.JPG /home/aistudio/work/dataset_xml_format/JPEGImages/
!mv /home/aistudio/work/dataset_xml_format/dataset_xml_format/*.xml /home/aistudio/work/dataset_xml_format/Annotations/

In order to facilitate subsequent processing, we can unify the suffix name of the picture.

In [ ]

%cd /home/aistudio/work/dataset_xml_format/JPEGImages/
!rename 's/\.jpg/\.png/'  ./*
!rename 's/\.JPG/\.png/'  ./*

Step03:  Divide the dataset

First install PaddleX.

In [ ]

!pip install paddlex

Then, we divide the training set and validation set according to the ratio of 0.7:0.3 through the split_dataset command in paddlex.

In [ ]

!paddlex --split_dataset --format VOC --dataset_dir /home/aistudio/work/dataset_xml_format --val_value 0.3

After division, we can see that there are three files train_list.txt, val_list.txt and labels.txt in the current path, which represent:

• Training set images and their annotation files
• Validation set images and their annotation files
• dataset label

3. Code implementation

3.1 Install Paddle Detection

In [ ]

# 克隆PaddleDetection仓库
#!git clone https://github.com/PaddlePaddle/PaddleDetection.git

# 安装其他依赖
%cd /home/aistudio/PaddleDetection/
!pip install -r requirements.txt

# 编译安装paddledet
!python setup.py install

3.2 Detection data analysis

Detection frame aspect ratio analysis:  By drawing the detection frame aspect ratio distribution histogram to reflect the distribution of the current detection frame aspect ratio.

In [ ]

import os
from unicodedata import name
import xml.etree.ElementTree as ET
import glob
import matplotlib.pyplot as plt

def ratio(indir):
    # 提取xml文件列表
    os.chdir(indir)
    annotations = os.listdir('.')
    annotations = glob.glob(str(annotations) + '*.xml')
    # count_0, count_1, count_2, count_3 = 0, 0, 0, 0 # 举反例，不要这么写
    count = [0 for i in range(20)]

    for i, file in enumerate(annotations): # 遍历xml文件
        # actual parsing
        in_file = open(file, encoding = 'utf-8')
        tree = ET.parse(in_file)
        root = tree.getroot()

        # 遍历文件的所有检测框
        for obj in root.iter('object'):
            xmin = obj.find('bndbox').find('xmin').text
            ymin = obj.find('bndbox').find('ymin').text
            xmax = obj.find('bndbox').find('xmax').text
            ymax = obj.find('bndbox').find('ymax').text
            Aspect_ratio = (int(ymax)-int(ymin)) / (int(xmax)-int(xmin))
            if int(Aspect_ratio/0.25) < 19:
                count[int(Aspect_ratio/0.25)] += 1
            else:
                count[-1] += 1
    sign = [0.25*i for i in range(20)]
    plt.bar(x=sign, height=count)
    plt.savefig("/home/aistudio/work/hw.png") 
    plt.show()
    print(count)

indir='/home/aistudio/work/dataset_xml_format/Annotations/'   # xml文件所在的目录
ratio(indir)

The result is as follows:

Image size analysis:  Through image size analysis, we can see that the images in this dataset are of different sizes.

In [ ]

import os
from unicodedata import name
import xml.etree.ElementTree as ET
import glob

def Image_size(indir):
    # 提取xml文件列表
    os.chdir(indir)
    annotations = os.listdir('.')
    annotations = glob.glob(str(annotations) + '*.xml')
    width_heights = []

    for i, file in enumerate(annotations): # 遍历xml文件
        # actual parsing
        in_file = open(file, encoding = 'utf-8')
        tree = ET.parse(in_file)
        root = tree.getroot()
        width = int(root.find('size').find('width').text)
        height = int(root.find('size').find('height').text)
        if [width, height] not in width_heights: width_heights.append([width, height])
    print("数据集中，有{}种不同的尺寸，分别是：".format(len(width_heights)))
    for item in width_heights:
        print(item)

indir='/home/aistudio/work/dataset_xml_format/Annotations/'   # xml文件所在的目录
Image_size(indir)

3.3 Model Training

Step01:  Move the dataset to the /home/aistudio/PaddleDetection/dataset directory.

In [13]

!mv /home/aistudio/work/dataset_xml_format /home/aistudio/PaddleDetection/dataset/

Step02:  Single card training

This project chooses Baidu Flying Paddle's self-developed model PP-YOLOE+. PP-YOLOE is an excellent single-stage anchor-free model based on PP-YOLOv2, which surpasses many popular YOLO models. PP-YOLOE has a series of models, namely s/m/l/x, which can be configured by width multiplier and depth multiplier. PP-YOLOE avoids the use of special operators such as Deformable Convolution or Matrix NMS, so that it can be easily deployed on a variety of hardware.

The PP-YOLOE model training process uses 8 GPUs for mixed precision training, and this project uses a single-card V100 during the training process, so it needs to follow the formula ����=��������∗(�� ����ℎ������∗������������)/(����ℎ����������∗������ ����������)lrnew​=lrdefault​∗(batchsizenew​∗GPUnumbernew​)/(batchsizedefault​∗GPUnumberdefault​) Adjust the learning rate to 1/8 of the original. At the same time, PP-YOLOE+ supports mixed precision training.

ERROR2: We can see that there is such a warning libpng warning: iCCP: known incorrect sRGB profile during the training process.

SOLUTION2: Resave after reading through skimage, the code is as follows.

In [ ]

!pip install scikit-image

In [ ]

import os
from tqdm import tqdm
import cv2
from skimage import io

path = r"/home/aistudio/PaddleDetection/dataset/dataset_xml_format/JPEGImages/"

fileList = os.listdir(path)
for i in tqdm(fileList):
    image = io.imread(path+i)
    image = cv2.cvtColor(image, cv2.COLOR_RGBA2BGRA)
    cv2.imencode('.png',image)[1].tofile(path+i)

After thirty rounds of iterations, we can see that the trained model has achieved good results in the verification set, with a mAP of 90.73%, which meets the standards of our project.

In [ ]

%cd /home/aistudio/PaddleDetection/
!python tools/train.py -c configs/ppyoloe/voc/ppyoloe_plus_crn_l_30e_voc.yml --eval --amp --use_vdl True --vdl_log_dir vdl_log_dir/scalar

The loss function is shown in the figure:

3.4 Model Evaluation

Evaluate our validation set on a single GPU with the following command.

In [ ]

!python tools/eval.py -c configs/ppyoloe/voc/ppyoloe_plus_crn_l_30e_voc.yml -o weights=output/ppyoloe_plus_crn_l_30e_voc/best_model.pdparams

3.5 Model Inference

We can infer all the images in the file on a single GPU with the following command.

In [ ]

!python tools/infer.py -c configs/ppyoloe/voc/ppyoloe_plus_crn_l_30e_voc.yml -o weights=output/ppyoloe_plus_crn_l_30e_voc/best_model.pdparams --infer_dir=dataset/dataset_xml_format/JPEGImages --output_dir infer_output/

3.6 Model export

PP-YOLOE+ needs to export the model through tools/export_model.py for deployment or speed test on GPU.

In [ ]

!python tools/export_model.py -c configs/ppyoloe/voc/ppyoloe_plus_crn_l_30e_voc.yml -o weights=output/ppyoloe_plus_crn_l_30e_voc/best_model.pdparams

3.7 FastDeploy rapid deployment

Environment preparation:  The main kit used in the deployment of this project is FastDeploy, the propeller deployment tool, so we install FastDeploy first.

In [ ]

!pip install fastdeploy-gpu-python -f https://www.paddlepaddle.org.cn/whl/fastdeploy.html

Deployment model:

Import the Flying Paddle deployment tool FastDepoy package and create a Runtime option. The specific implementation is shown in the following code.

In [34]

import fastdeploy as fd
import cv2
import os

In [35]

def build_option(device='cpu', use_trt=False):
    option = fd.RuntimeOption()

    if device.lower() == "gpu":
        option.use_gpu()

    if use_trt:
        option.use_trt_backend()
        option.set_trt_input_shape("image", [1, 3, 640, 640])
        option.set_trt_input_shape("scale_factor", [1, 2])

    return option

Configure the model path, create a Runtime option, and specify the deployment device and backend inference engine. The code implementation is as follows.

In [ ]

# 配置模型路径
model_path = '/home/aistudio/PaddleDetection/output_inference/ppyoloe_plus_crn_l_30e_voc'
image_path = '/home/aistudio/PaddleDetection/dataset/dataset_xml_format/JPEGImages/foto00262.png'
model_file = os.path.join(model_path, "model.pdmodel")
params_file = os.path.join(model_path, "model.pdiparams")
config_file = os.path.join(model_path, "infer_cfg.yml")

# 创建RuntimeOption
runtime_option = build_option(device='gpu', use_trt=False)

# 创建PPYOLOE+模型
model = fd.vision.detection.PPYOLO(model_file,
                                   params_file,
                                   config_file,
                                   runtime_option=runtime_option)

# 预测图片检测结果
im = cv2.imread(image_path)
result = model.predict(im.copy())
print(result)

# 预测结果可视化
vis_im = fd.vision.vis_detection(im, result, score_threshold=0.5)
cv2.imwrite("/home/aistudio/work/visualized_result.jpg", vis_im)
print("Visualized result save in ./visualized_result.jpg")

The reasoning results are as follows:

4. Effect display

Some visualization results are as follows: