PPLiteSeg real-time semantic segmentation prediction results output control unmanned vehicle steering angle direction to realize unmanned driving along the lane

I. Introduction

Following the above, the training of the data set and the modification of the framework have been completed to realize the prediction results of the real-time output mask, but it is impossible to realize unmanned driving only by doing these.

PPLiteSeg trains its own data set to achieve automatic driving and transforms it into an API that can be called by other Python programs for real-time semantic segmentation (ultra-low latency) - Leonard2021's Blog - CSDN Blog

A Tutorial for Using Baidu Flying EISeg Efficient Interactive Labeling and Segmentation Software_Leonard2021's Blog-CSDN Blog

The overall idea: the output prediction mask of real-time semantic segmentation, independently read the lane line segmentation result mask in the overall prediction segmentation, and then perform image binarization Canny edge detection through opencv to obtain the edge of the lane line, and then pass the HoughLinesP Huo The line change obtains a large number of scattered coordinates on the lane line. After obtaining the midpoint of the starting point and the end point of the lane line in these scattered coordinates, set the logic to judge whether the lane line leads to the front, left, or right, and obtain the output lane For the angle between the midpoint of the area and the midpoint of the starting lane area, set the logic to synchronize the output angle to the direction angle control of the front steering wheel of the unmanned vehicle, so that the car can drive automatically along the lane.

The dataset looks like this:

 

Two, actual combat

The part of the training function, which I mentioned in the previous article, will not be repeated here, and I will directly start modifying and using the calling part.

Create a new visualize_myself.py:

# Copyright (c) 2020 PaddlePaddle Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

import os

import cv2
import numpy as np
from PIL import Image as PILImage


def visualize(image, result, color_map, save_dir=None, weight=0.6):
    """
    Convert predict result to color image, and save added image.

    Args:
        image (str): The path of origin image.
        result (np.ndarray): The predict result of image.
        color_map (list): The color used to save the prediction results.
        save_dir (str): The directory for saving visual image. Default: None.
        weight (float): The image weight of visual image, and the result weight is (1 - weight). Default: 0.6

    Returns:
        vis_result (np.ndarray): If `save_dir` is None, return the visualized result.
    """

    color_map = [color_map[i:i + 3] for i in range(0, len(color_map), 3)]
    color_map = np.array(color_map).astype("uint8")
    # Use OpenCV LUT for color mapping
    c1 = cv2.LUT(result, color_map[:, 0])
    c2 = cv2.LUT(result, color_map[:, 1])
    c3 = cv2.LUT(result, color_map[:, 2])
    pseudo_img = np.dstack((c3, c2, c1))

    #im = cv2.imread(image)
    im = image.copy()
    vis_result = cv2.addWeighted(im, weight, pseudo_img, 1 - weight, 0)

    if save_dir is not None:
        if not os.path.exists(save_dir):
            os.makedirs(save_dir)
        image_name = os.path.split(image)[-1]
        out_path = os.path.join(save_dir, image_name)
        cv2.imwrite(out_path, vis_result)
    else:
        return vis_result


def get_pseudo_color_map(pred, color_map=None):
    """
    Get the pseudo color image.

    Args:
        pred (numpy.ndarray): the origin predicted image.
        color_map (list, optional): the palette color map. Default: None,
            use paddleseg's default color map.

    Returns:
        (numpy.ndarray): the pseduo image.
    """
    pred_mask = PILImage.fromarray(pred.astype(np.uint8), mode='P')
    if color_map is None:
        color_map = get_color_map_list(256)
    pred_mask.putpalette(color_map)
    #print(len(pred_mask.split()), type(pred_mask))
    #image = pred_mask.convert("RGB")#单通道转化为3通道
    #img = cv2.cvtColor(np.asarray(image), cv2.COLOR_BGR2GRAY)#转化为cv的格式返回
    return pred_mask


def get_color_map_list(num_classes, custom_color=None):
    """
    Returns the color map for visualizing the segmentation mask,
    which can support arbitrary number of classes.

    Args:
        num_classes (int): Number of classes.
        custom_color (list, optional): Save images with a custom color map. Default: None, use paddleseg's default color map.

    Returns:
        (list). The color map.
    """

    num_classes += 1
    color_map = num_classes * [0, 0, 0]
    for i in range(0, num_classes):
        j = 0
        lab = i
        while lab:
            color_map[i * 3] |= (((lab >> 0) & 1) << (7 - j))
            color_map[i * 3 + 1] |= (((lab >> 1) & 1) << (7 - j))
            color_map[i * 3 + 2] |= (((lab >> 2) & 1) << (7 - j))
            j += 1
            lab >>= 3
    color_map = color_map[3:]

    if custom_color:
        color_map[:len(custom_color)] = custom_color
    return color_map


def paste_images(image_list):
    """
    Paste all image to a image.
    Args:
        image_list (List or Tuple): The images to be pasted and their size are the same.
    Returns:
        result_img (PIL.Image): The pasted image.
    """
    assert isinstance(image_list,
                      (list, tuple)), "image_list should be a list or tuple"
    assert len(
        image_list) > 1, "The length of image_list should be greater than 1"

    pil_img_list = []
    for img in image_list:
        if isinstance(img, str):
            assert os.path.exists(img), "The image is not existed: {}".format(
                img)
            img = PILImage.open(img)
            img = np.array(img)
        elif isinstance(img, np.ndarray):
            img = PILImage.fromarray(img)
        pil_img_list.append(img)

    sample_img = pil_img_list[0]
    size = sample_img.size
    for img in pil_img_list:
        assert size == img.size, "The image size in image_list should be the same"

    width, height = sample_img.size
    result_img = PILImage.new(sample_img.mode,
                              (width * len(pil_img_list), height))
    for i, img in enumerate(pil_img_list):
        result_img.paste(img, box=(width * i, 0))

    return result_img

New predict_with_api.py : The modified predict is made into an API interface that can run in real time

import cv2
import numpy as np
import paddle
from paddleseg.core import infer
from paddleseg.utils import visualize
import visualize_myself
from PIL import Image as PILImage


def preprocess(im_path, transforms):
    data = {}
    data['img'] = im_path
    data = transforms(data)
    data['img'] = data['img'][np.newaxis, ...]
    data['img'] = paddle.to_tensor(data['img'])
    return data

def predict(model,
            model_path,
            transforms,
            image_list,
            aug_pred=False,
            scales=1.0,
            flip_horizontal=True,
            flip_vertical=False,
            is_slide=False,
            stride=None,
            crop_size=None,
            custom_color=None
            ):
    # 加载模型权重
    para_state_dict = paddle.load(model_path)
    model.set_dict(para_state_dict)
    # 设置模型为评估模式
    model.eval()
    # 读取图像
    im = image_list.copy()
    color_map = visualize.get_color_map_list(256, custom_color=custom_color)
    with paddle.no_grad():
        data = preprocess(im, transforms)
        # 是否开启多尺度翻转预测
        if aug_pred:
            pred, _ = infer.aug_inference(
                model,
                data['img'],
                trans_info=data['trans_info'],
                scales=scales,
                flip_horizontal=flip_horizontal,
                flip_vertical=flip_vertical,
                is_slide=is_slide,
                stride=stride,
                crop_size=crop_size)
        else:
            pred, _ = infer.inference(
                model,
                data['img'],
                trans_info=data['trans_info'],
                is_slide=is_slide,
                stride=stride,
                crop_size=crop_size)
        # 将返回数据去除多余的通道，并转为uint8类型，方便保存为图片

        pred = paddle.squeeze(pred)
        pred = pred.numpy().astype('uint8')

        # 展示结果
        added_image = visualize_myself.visualize(image= im,result= pred,color_map=color_map, weight=0.6)
        #cv2.imshow('image_predict', added_image)

        # save pseudo color prediction
        pred_mask = visualize_myself.get_pseudo_color_map(pred, color_map)
        #cv2.waitKey(0)
        #cv2.destroyAllWindows()
        return added_image,pred_mask

Create a new seg_read.py: The main function is to change the output of the overall prediction to specify the output of a certain type of segmentation results. Here I mainly only output the prediction part of the lane line segmentation

from PIL import Image
import numpy as np
import cv2

#生成染色版
def get_voc_palette(num_classes):
    n = num_classes
    palette = [0]*(n*3)
    for j in range(0,n):
            lab = j
            palette[j*3+0] = 0
            palette[j*3+1] = 0
            palette[j*3+2] = 0
            i = 0
            while (lab > 0):
                    palette[j*3+0] |= (((lab >> 0) & 1) << (7-i))
                    palette[j*3+1] |= (((lab >> 1) & 1) << (7-i))
                    palette[j*3+2] |= (((lab >> 2) & 1) << (7-i))
                    i = i + 1
                    lab >>= 3
    return palette

#染色代码
def colorize_mask(mask, palette):
    zero_pad = 256 * 3 - len(palette)
    for i in range(zero_pad):
                    palette.append(0)
    new_mask = Image.fromarray(mask.astype(np.uint8)).convert('P')
    new_mask.putpalette(palette)
    return new_mask

#生成指定类别的语义分割预测结果
def generate_img_with_choice_class(img,classes:list,num_classes:int):
    #传入 图像路径 和 需要预测的类别  总共的类别
    #img = Image.open(img)#

    img = np.asarray(img)
    f_img = img.copy()
    for idx,c in enumerate(classes):
        f_img[np.where(img==c)] = 0 #将对应位置置零
    f_img = colorize_mask(f_img,get_voc_palette(num_classes)) # 进行染色处理
    image = f_img.convert("RGB")
    #image.save('output/process_img.png')
    img = cv2.cvtColor(np.asarray(image), cv2.COLOR_BGR2GRAY)
    return img

if __name__ == '__main__':
    ''''''
    img_path = r'data/annotations/000026.png'
    choice_list = [0,2]
    img = generate_img_with_choice_class(img_path,choice_list,3)
    cv2.imshow('image',img)
    #cv2.imwrite('output/3.jpg', img)
    #cv2.waitKey(0)

New opencv manual threshold segmentation.py: manually adjust the threshold of canny edge detection through the slider

import cv2
#载入图片
img_original=cv2.imread('output/1.jpg')
#设置窗口
cv2.namedWindow('Canny')
#定义回调函数
def nothing(x):
    pass
#创建两个滑动条，分别控制threshold1，threshold2
cv2.createTrackbar('threshold1','Canny',50,400,nothing)
cv2.createTrackbar('threshold2','Canny',100,400,nothing)
while(1):
    #返回滑动条所在位置的值
    threshold1=cv2.getTrackbarPos('threshold1','Canny')
    threshold2=cv2.getTrackbarPos('threshold2','Canny')
    #Canny边缘检测
    img_edges=cv2.Canny(img_original,threshold1,threshold2)
    #显示图片
    cv2.imshow('original',img_original)
    cv2.imshow('Canny',img_edges)
    if cv2.waitKey(1)==ord('q'):
        break
cv2.destroyAllWindows()

Create a new HoughLines_return.py: the main function is to extract a large number of scattered coordinates on the lane line through canny feature detection, and then extract the output coordinates of the four closest edges (including: the top, the left, the bottom, and the right) , get the midpoint coordinates of the 4 directions on average. If the midpoint of the leftmost lane line edge is close to the upper left side and does not touch the upper side, the output angle is the angle between the midpoint of the left edge and the midpoint of the lower edge. , and the same for the right and top. After the output, the direction angle of the front steering wheel of the unmanned vehicle can be synchronized in real time through serial communication.

import cv2
import matplotlib.pyplot as plt
import numpy as np
from collections import Counter
import math

def sort_max(list):
    list1 = list.copy()#复制一份，不破坏原来的列表
    list1.sort()#从小到大排序
    max1 = list1[-1]#最大
    max2 = list1[-2]#第二大
    max3 = list1[-3]#第3大
    max4 = list1[-4]#第4大
    return max1,max2,max3,max4

def sort_min(list):
    list1 = list.copy()  # 复制一份，不破坏原来的列表
    list1.sort()  # 从小到大排序
    min1 = list1[0]  # 最大
    min2 = list1[1]  # 第二大
    min3 = list1[2]  # 第3大
    min4 = list1[3]  # 第4大
    return min1, min2, min3, min4

def azimuthangle(x1, y1, x2, y2):
    """ 已知两点坐标计算角度 -
    :param x1: 原点横坐标值
    :param y1: 原点纵坐标值
    :param x2: 目标点横坐标值
    :param y2: 目标纵坐标值
    """
    angle = 0.0
    dx = x2 - x1
    dy = y2 - y1
    if x2 == x1:
        angle = math.pi / 2.0
        if y2 == y1:
            angle = 0.0
        elif y2 < y1:
            angle = 3.0 * math.pi / 2.0
    elif x2 > x1 and y2 > y1:
        angle = math.atan(dx / dy)
    elif x2 > x1 and y2 < y1:
        angle = math.pi / 2 + math.atan(-dy / dx)
    elif x2 < x1 and y2 < y1:
        angle = math.pi + math.atan(dx / dy)
    elif x2 < x1 and y2 > y1:
        angle = 3.0 * math.pi / 2.0 + math.atan(dy / -dx)
    return angle * 180 / math.pi


def get_index(l, x, n):
    # 函数作用: 获取某个元素第n次出现在列表的下标
    # 参数列表: 第一个参数为可迭代对象, 第二个参数为要查找的数, 第三个参数为要查找第几个出现的x
    l_count = l.count(x)
    result = None
    if n <= l_count:
        num = 0
        for item in enumerate(l):
            if item[1] == x:
                num += 1
            if num == n:
                result = item[0]
                break
    else:
        print("列表里总共有{}个{}".format(l_count, x))
    return result

def get_top_mid(x_list,y_list,judge):#真为上方，假为下方
    if judge ==True:
        min1_y, min2_y, min3_y, min4_y = sort_min(y_list)
    else:
        min1_y, min2_y, min3_y, min4_y = sort_max(y_list)
    #print("min1:", min1_y, "min2:", min2_y, "min3:", min3_y, "min4:", min4_y)
    min_list_y = [min1_y, min2_y, min3_y, min4_y]
    b = dict(Counter(min_list_y))
    same = [key for key, value in b.items() if value > 1]  # 只展示重复元素
    min_list_x = []
    i_1 = 1
    i_2 = 1
    for max in min_list_y:
        if len(same)==0:
            y_index = get_index(y_list, max, 1)
        elif len(same)==1:
            if max == same[0]:
                y_index = get_index(y_list, max, i_1)
                i_1 =i_1+1
            else:
                y_index = get_index(y_list, max, 1)
        elif len(same)==2:
            if max == same[0]:
                y_index = get_index(y_list, max, i_1)
                i_1 =i_1+1
            elif max == same[1]:
                y_index = get_index(y_list, max, i_2)
                i_2 =i_2+1
            else:
                y_index = get_index(y_list, max, 1)
        min_list_x.append(x_list[y_index])
    #print("min_list_x", min_list_x)
    top_mid = [sum(min_list_x) / len(min_list_x), sum(min_list_y) / len(min_list_y)]
    #print("top_mid:", top_mid)
    return top_mid

def get_side_mid(x_list,y_list,judge):#真为左，假为右
    if judge ==True:
        m1_x, m2_x, m3_x, m4_x = sort_min(x_list)
    else:
        m1_x, m2_x, m3_x, m4_x = sort_max(x_list)
    #print("min1:", m1_x, "min2:", m2_x, "min3:", m3_x, "min4:", m4_x)
    min_list_x = [m1_x, m2_x, m3_x, m4_x]
    b = dict(Counter(min_list_x))
    same = [key for key, value in b.items() if value > 1]  # 只展示重复元素
    #print("same:", same, len(same))
    min_list_y = []
    i_1 = 1
    i_2 = 1
    for max in min_list_x:
        if len(same)==0:
            x_index = get_index(x_list, max, 1)
        elif len(same) == 1:
            if max == same:
                x_index = get_index(x_list, max, i_1)
                i_1 = i_1 + 1
            else:
                x_index = get_index(x_list, max, 1)
            #print("x_index_1:", type(x_index), x_index)
        elif len(same)==2:
            if max == same[0]:
                x_index = get_index(x_list, max, i_1)
                i_1 = i_1 + 1
            elif max == same[1]:
                x_index = get_index(x_list, max, i_2)
                i_2 = i_2 + 1
            else:
                x_index = get_index(x_list, max, 1)
            #print("x_index_1:", type(x_index), x_index)
        min_list_y.append(y_list[x_index])
    #print("min_list_x", min_list_y)
    side_mid = [sum(min_list_x) / len(min_list_x), sum(min_list_y) / len(min_list_y)]
    #print("side_mid:", side_mid)
    return side_mid

def HL_return(src):
    img = src.copy()
    # 二值化图像（Canny边缘检测）
    # gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)
    dst_img = cv2.Canny(img, 11, 16)

    cv2.imshow('Canny', dst_img)
    #cv2.waitKey(0)

    lines = cv2.HoughLinesP(dst_img, 1, np.pi / 180, 20)

    x_list = []
    y_list = []

    for line in lines:
        for x1, y1, x2, y2 in line:
            x_list.append(x1)
            x_list.append(x2)
            y_list.append(y1)
            y_list.append(y2)
            cv2.line(img, (x1, y1), (x2, y2), (0, 255, 0), 3)
        cv2.imshow("HoughLinesP", img)
        #cv2.waitKey(0)

    top_mid = get_top_mid(x_list,y_list,True)#得到上面的中点
    down_mid = get_side_mid(x_list,y_list,False)#得到下面的中点
    left_side_mid = get_side_mid(x_list,y_list,True)#得到左边的中点
    right_side_mid = get_side_mid(x_list,y_list,False)#得到右边的中点
    print("top_mid:",top_mid,"dowm_mid:",down_mid,"left_mid:",left_side_mid,"right_mid:",right_side_mid)

    sp = src.shape
    sz1 = sp[0]  # height(rows) of image
    sz2 = sp[1]  # width(colums) of image
    sz3 = sp[2]  # the pixels value is made up of three primary colors
    if left_side_mid[1]<(sp[0]*0.5):
        angle =azimuthangle (left_side_mid[0],left_side_mid[1],down_mid[0],down_mid[1])
    elif right_side_mid[1]<(sp[0]*0.5):
        angle = azimuthangle(right_side_mid[0], right_side_mid[1], down_mid[0], down_mid[1])
    else:
        angle = azimuthangle(top_mid[0], top_mid[1], down_mid[0], down_mid[1])
    #print("angle:", type(angle), angle)
    #cv2.waitKey(0)
    return angle

if __name__ == '__main__':
    src = cv2.imread(r"D:\pyCharmdata\PPLiteSeg_demo\output\2.jpg")
    angle =HL_return(src)
    if (angle - 90) > 0:
        print("请右转:", math.fabs(angle - 90), "度")
    elif (angle - 90) < 0:
        print("请左转:", math.fabs(angle - 90), "度")
    elif angle == 90:
        print("正在保持90度直行")

 Create a new demo_run_API.py: call the real-time segmentation API and output the steering angle in real time

import cv2
from predict_with_api import predict
from paddleseg.models import PPLiteSeg
from paddleseg.models.backbones import STDC1
import paddleseg.transforms as T
from seg_read import generate_img_with_choice_class
from HoughLines_return import HL_return
import math

backbone = STDC1()

model = PPLiteSeg(num_classes=13,
                 backbone= backbone,
                 arm_out_chs = [32, 64, 128],
                 seg_head_inter_chs = [32, 64, 64],
                 pretrained=None)

transforms = T.Compose([
    T.Resize(target_size=(512, 512)),
    T.RandomHorizontalFlip(),
    T.Normalize()
])
model_path = 'output/best_model/model.pdparams'
cap=cv2.VideoCapture(0)# 0

if __name__ == '__main__':
    while True:
        rec,img = cap.read()
        added_image,pred_mask = predict(model=model,model_path=model_path, transforms=transforms,image_list=img)
        cv2.imshow('image_predict', added_image)#显示原图像与分割预测的合并图
        #
        choice_list = [0, 2]#只展示第2个类型的分割结果
        img = generate_img_with_choice_class(pred_mask, choice_list, 3)# PIL图像输入、展示第几个类型的分割、总的分割种类
        cv2.imshow('image', img)#将返回的指定分割类型的灰度图显示出来
        angle = HL_return(img)
        if (angle - 90) > 0:
            print("请右转:", math.fabs(angle - 90), "度")
        elif (angle - 90) < 0:
            print("请左转:", math.fabs(angle - 90), "度")
        elif angle == 90:
            print("正在保持90度直行")

        if cv2.waitKey(1)==ord('q'):
            break

3. Summary

In general, although the overall development work has been completed, due to the use of data sets, and the number of data sets used is small, the degree of training is low. The experiment has not yet been implemented, and there may be many debugging and logical problems to be discovered and resolved, and the threshold for judging the flow direction of the lane line is estimated manually, and the actual threshold difference is open to question. In terms of logic control, compared with the algorithms of big manufacturers, it seems less mature and simple. The part that synchronizes the output angle with the direction angle of the steering wheel uses pyserial to communicate with the lower computer, and the lower computer then controls the steering wheel of the motor to the specified angle.

Coupled with the target detection part, a simple unmanned vision part is formed.

Explosively change the detect.py of YOLOV7 and make it into an API interface for other python programs to call (ultra-low latency)_Leonard2021's Blog-CSDN Blog

I am completely independent and self-made gadget, the simple version of the unmanned driving visual part, the accuracy is definitely not as good as the finished algorithm of the big factory, if I have time and energy, I will continue to improve and update it.

The structure of this project can be taken by yourself if necessary:

PPLiteSeg_AutoDrive_demo.zip-Deep Learning Documentation Resources-CSDN Download

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

This is the end of this article, if it is helpful to you, welcome to click three times! ! !