Smart traffic day03-Lane line detection implementation 06: Lane line positioning and fitting + code implementation

learning target

• Understand the idea of ​​the histogram to determine the location of the lane lines

---

According to the previously detected lane line information, we use the method of histogram and sliding window to accurately locate the lane line and perform fitting.

1. Positioning thinking

The following image is the result of our detected lane lines:

Count the number of white pixels in each column along the x-axis direction, the abscissa is the number of columns in the image, and the ordinate represents the number of white points in each column, then this picture is a "histogram", as shown in the following figure:

Comparing the above two figures, it can be found that the number of columns corresponding to the maximum value on the left half of the histogram is the position of the left lane line, and the number of columns corresponding to the maximum value on the right half of the histogram is the position of the right lane line.

After determining the approximate positions of the left and right lane lines, use the "sliding window" method to search for the points of the left and right lane lines in the figure.

Sliding window search process:

• Set the size of the search window (width and height): In general, the width is manually set, and the height is calculated by dividing the image size by the number of set search windows.

• Take the search starting point as the base point of the current search, and take the current base point as the center to do a grid search.

• 

• Make horizontal and vertical histogram statistics for each search window, count the number of non-zero pixels in the search box area, and filter out boxes with less than 50 non-zero pixels.

• Calculate the mean value of non-zero pixel coordinates as the center of the current search box, and perform a second-order polynomial fit on these center points to obtain the lane line curve parameters corresponding to the current search.

2. Realize

The code to detect lane lines using histogram and sliding window is as follows:

def cal_line_param(binary_warped):
    # 1.确定左右车道线的位置
    # 统计直方图
    histogram = np.sum(binary_warped[:, :], axis=0)
    # 在统计结果中找到左右最大的点的位置，作为左右车道检测的开始点
    # 将统计结果一分为二，划分为左右两个部分，分别定位峰值位置，即为两条车道的搜索位置
    midpoint = np.int(histogram.shape[0] / 2)
    leftx_base = np.argmax(histogram[:midpoint])
    rightx_base = np.argmax(histogram[midpoint:]) + midpoint
    # 2.滑动窗口检测车道线
    # 设置滑动窗口的数量，计算每一个窗口的高度
    nwindows = 9
    window_height = np.int(binary_warped.shape[0] / nwindows)
    # 获取图像中不为0的点
    nonzero = binary_warped.nonzero()
    nonzeroy = np.array(nonzero[0])
    nonzerox = np.array(nonzero[1])
    # 车道检测的当前位置
    leftx_current = leftx_base
    rightx_current = rightx_base
    # 设置x的检测范围，滑动窗口的宽度的一半，手动指定
    margin = 100
    # 设置最小像素点，阈值用于统计滑动窗口区域内的非零像素个数，小于50的窗口不对x的中心值进行更新
    minpix = 50
    # 用来记录搜索窗口中非零点在nonzeroy和nonzerox中的索引
    left_lane_inds = []
    right_lane_inds = []

    # 遍历该副图像中的每一个窗口
    for window in range(nwindows):
        # 设置窗口的y的检测范围，因为图像是（行列）,shape[0]表示y方向的结果，上面是0
        win_y_low = binary_warped.shape[0] - (window + 1) * window_height
        win_y_high = binary_warped.shape[0] - window * window_height
        # 左车道x的范围
        win_xleft_low = leftx_current - margin
        win_xleft_high = leftx_current + margin
        # 右车道x的范围
        win_xright_low = rightx_current - margin
        win_xright_high = rightx_current + margin

        # 确定非零点的位置x,y是否在搜索窗口中，将在搜索窗口内的x,y的索引存入left_lane_inds和right_lane_inds中
        good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &
                          (nonzerox >= win_xleft_low) & (nonzerox < win_xleft_high)).nonzero()[0]
        good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &
                           (nonzerox >= win_xright_low) & (nonzerox < win_xright_high)).nonzero()[0]
        left_lane_inds.append(good_left_inds)
        right_lane_inds.append(good_right_inds)

        # 如果获取的点的个数大于最小个数，则利用其更新滑动窗口在x轴的位置
        if len(good_left_inds) > minpix:
            leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
        if len(good_right_inds) > minpix:
            rightx_current = np.int(np.mean(nonzerox[good_right_inds]))

    # 将检测出的左右车道点转换为array
    left_lane_inds = np.concatenate(left_lane_inds)
    right_lane_inds = np.concatenate(right_lane_inds)

    # 获取检测出的左右车道点在图像中的位置
    leftx = nonzerox[left_lane_inds]
    lefty = nonzeroy[left_lane_inds]
    rightx = nonzerox[right_lane_inds]
    righty = nonzeroy[right_lane_inds]

    # 3.用曲线拟合检测出的点,二次多项式拟合，返回的结果是系数
    left_fit = np.polyfit(lefty, leftx, 2)
    right_fit = np.polyfit(righty, rightx, 2)
    return left_fit, right_fit

The fitting result of lane line detection is shown in the following figure:

The green box is the result of the sliding window, and the yellow line in the middle is the result of the lane line fitting.

Next, we will draw the lane area, that is, draw a polygon in the middle of the detected lane line. The code is as follows:

def fill_lane_poly(img, left_fit, right_fit):
    # 获取图像的行数
    y_max = img.shape[0]
    # 设置输出图像的大小，并将白色位置设为255
    out_img = np.dstack((img, img, img)) * 255
    # 在拟合曲线中获取左右车道线的像素位置
    left_points = [[left_fit[0] * y ** 2 + left_fit[1] * y + left_fit[2], y] for y in range(y_max)]
    right_points = [[right_fit[0] * y ** 2 + right_fit[1] * y + right_fit[2], y] for y in range(y_max - 1, -1, -1)]
    # 将左右车道的像素点进行合并
    line_points = np.vstack((left_points, right_points))
    # 根据左右车道线的像素位置绘制多边形
    cv2.fillPoly(out_img, np.int_([line_points]), (0, 255, 0))
    return out_img

Results as shown below:

Two of the methods will be introduced to you:

np.vstack: stack arrays vertically (row order) to form a new array

np.dstack: stack arrays horizontally (column order) to form a new array

Interpretation of np.hstack(), np.vstack() /lllxxq141592654/article/details/84260091 https://blog.csdn.net/lllxxq141592654/article/details/84260091

An example is as follows:

Backproject the detected lanes to the original image and directly call the perspective transformation method:

transform_img_inverse = img_perspect_transform(result, M_inverse)

Results as shown below:

Finally, superimpose it on the original image, there is:

Summarize:

Know the idea of ​​precise positioning of lane lines:

First, use the histogram method to determine the position of the left and right lane lines, then use the sliding window method to search for the position of the lane lines, and perform fitting, then draw the lane area, and perform back-projection to obtain the lane area in the original image.

---

Code:

# encoding:utf-8

from matplotlib import font_manager
my_font = font_manager.FontProperties(fname="/System/Library/Fonts/PingFang.ttc")

import cv2
import numpy as np
import matplotlib.pyplot as plt
#遍历文件夹
import glob
from moviepy.editor import VideoFileClip
import sys
reload(sys)
sys.setdefaultencoding('utf-8')



"""参数设置"""
nx = 9
ny = 6
#获取棋盘格数据
file_paths = glob.glob("./camera_cal/calibration*.jpg")


#相机矫正使用opencv封装好的api
#目的:得到内参、外参、畸变系数
def cal_calibrate_params(file_paths):
    #存储角点数据的坐标
    object_points = [] #角点在真实三维空间的位置
    image_points = [] #角点在图像空间中的位置
    #生成角点在真实世界中的位置
    objp = np.zeros((nx*ny,3),np.float32)
    #以棋盘格作为坐标，每相邻的黑白棋的相差1
    objp[:,:2] = np.mgrid[0:nx,0:ny].T.reshape(-1,2)
    #角点检测
    for file_path in file_paths:
        img = cv2.imread(file_path)
        #将图像灰度化
        gray = cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)
        #角点检测
        rect,coners = cv2.findChessboardCorners(gray,(nx,ny),None)

        #若检测到角点，则进行保存 即得到了真实坐标和图像坐标
        if rect == True :
            object_points.append(objp)
            image_points.append(coners)
    # 相机较真
    ret, mtx, dist, rvecs, tvecs = cv2.calibrateCamera(object_points, image_points, gray.shape[::-1], None, None)
    return ret, mtx, dist, rvecs, tvecs

# 图像去畸变：利用相机校正的内参，畸变系数
def img_undistort(img, mtx, dist):
    dis = cv2.undistort(img, mtx, dist, None, mtx)
    return dis

#车道线提取
#颜色空间转换--》边缘检测--》颜色阈值--》并且使用L通道进行白色的区域进行抑制
def pipeline(img,s_thresh = (170,255),sx_thresh=(40,200)):
    # 复制原图像
    img = np.copy(img)
    # 颜色空间转换
    hls = cv2.cvtColor(img,cv2.COLOR_RGB2HLS).astype(np.float)
    l_chanel = hls[:,:,1]
    s_chanel = hls[:,:,2]
    #sobel边缘检测
    sobelx = cv2.Sobel(l_chanel,cv2.CV_64F,1,0)
    #求绝对值
    abs_sobelx = np.absolute(sobelx)
    #将其转换为8bit的整数
    scaled_sobel = np.uint8(255 * abs_sobelx / np.max(abs_sobelx))
    #对边缘提取的结果进行二值化
    sxbinary = np.zeros_like(scaled_sobel)
    #边缘位置赋值为1，非边缘位置赋值为0
    sxbinary[(scaled_sobel >= sx_thresh[0]) & (scaled_sobel <= sx_thresh[1])] = 1

    #对S通道进行阈值处理
    s_binary = np.zeros_like(s_chanel)
    s_binary[(s_chanel >= s_thresh[0]) & (s_chanel <= s_thresh[1])] = 1

    # 结合边缘提取结果和颜色通道的结果，
    color_binary = np.zeros_like(sxbinary)
    color_binary[((sxbinary == 1) | (s_binary == 1)) & (l_chanel > 100)] = 1
    return color_binary

#透视变换-->将检测结果转换为俯视图。
#获取透视变换的参数矩阵【二值图的四个点】
def cal_perspective_params(img,points):
    # x与y方向上的偏移
    offset_x = 330
    offset_y = 0
    #转换之后img的大小
    img_size = (img.shape[1],img.shape[0])
    src = np.float32(points)
    #设置俯视图中的对应的四个点 左上角 右上角 左下角 右下角
    dst = np.float32([[offset_x, offset_y], [img_size[0] - offset_x, offset_y],
                      [offset_x, img_size[1] - offset_y], [img_size[0] - offset_x, img_size[1] - offset_y]])
    ## 原图像转换到俯视图
    M = cv2.getPerspectiveTransform(src, dst)
    # 俯视图到原图像
    M_inverse = cv2.getPerspectiveTransform(dst, src)
    return M, M_inverse

#根据透视变化矩阵完成透视变换
def img_perspect_transform(img,M):
    #获取图像大小
    img_size = (img.shape[1],img.shape[0])
    #完成图像的透视变化
    return cv2.warpPerspective(img,M,img_size)

# 精确定位车道线
#传入已经经过边缘检测的图像阈值结果的二值图，再进行透明变换
def cal_line_param(binary_warped):
    #定位车道线的大致位置==计算直方图
    histogram = np.sum(binary_warped[:,:],axis=0) #计算y轴
    # 将直方图一分为二，分别进行左右车道线的定位
    midpoint = np.int(histogram.shape[0]/2)
    #分别统计左右车道的最大值
    midpoint = np.int(histogram.shape[0] / 2)
    leftx_base = np.argmax(histogram[:midpoint]) #左车道
    rightx_base = np.argmax(histogram[midpoint:]) + midpoint #右车道
    #设置滑动窗口
    #对每一个车道线来说 滑动窗口的个数
    nwindows = 9
    #设置滑动窗口的高
    window_height = np.int(binary_warped.shape[0]/nwindows)
    #设置滑动窗口的宽度==x的检测范围,即滑动窗口的一半
    margin = 100
    #统计图像中非0点的个数
    nonzero = binary_warped.nonzero()
    nonzeroy = np.array(nonzero[0])#非0点的位置-x坐标序列
    nonzerox = np.array(nonzero[1])#非0点的位置-y坐标序列
    #车道检测位置
    leftx_current = leftx_base
    rightx_current = rightx_base
    #设置阈值：表示当前滑动窗口中的非0点的个数
    minpix = 50
    #记录窗口中，非0点的索引
    left_lane_inds = []
    right_lane_inds = []

    #遍历滑动窗口
    for window in range(nwindows):
        # 设置窗口的y的检测范围，因为图像是（行列）,shape[0]表示y方向的结果，上面是0
        win_y_low = binary_warped.shape[0] - (window + 1) * window_height #y的最低点
        win_y_high = binary_warped.shape[0] - window * window_height #y的最高点
        # 左车道x的范围
        win_xleft_low = leftx_current - margin
        win_xleft_high = leftx_current + margin
        # 右车道x的范围
        win_xright_low = rightx_current - margin
        win_xright_high = rightx_current + margin

        # 确定非零点的位置x,y是否在搜索窗口中，将在搜索窗口内的x,y的索引存入left_lane_inds和right_lane_inds中
        good_left_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &
                          (nonzerox >= win_xleft_low) & (nonzerox < win_xleft_high)).nonzero()[0]
        good_right_inds = ((nonzeroy >= win_y_low) & (nonzeroy < win_y_high) &
                           (nonzerox >= win_xright_low) & (nonzerox < win_xright_high)).nonzero()[0]
        left_lane_inds.append(good_left_inds)
        right_lane_inds.append(good_right_inds)

        # 如果获取的点的个数大于最小个数，则利用其更新滑动窗口在x轴的位置=修正车道线的位置
        if len(good_left_inds) > minpix:
            leftx_current = np.int(np.mean(nonzerox[good_left_inds]))
        if len(good_right_inds) > minpix:
            rightx_current = np.int(np.mean(nonzerox[good_right_inds]))

    # 将检测出的左右车道点转换为array
    left_lane_inds = np.concatenate(left_lane_inds)
    right_lane_inds = np.concatenate(right_lane_inds)

    # 获取检测出的左右车道x与y点在图像中的位置
    leftx = nonzerox[left_lane_inds]
    lefty = nonzeroy[left_lane_inds]
    rightx = nonzerox[right_lane_inds]
    righty = nonzeroy[right_lane_inds]

    # 3.用曲线拟合检测出的点,二次多项式拟合，返回的结果是系数
    left_fit = np.polyfit(lefty, leftx, 2)
    right_fit = np.polyfit(righty, rightx, 2)
    return left_fit, right_fit

#填充车道线之间的多边形
def fill_lane_poly(img,left_fit,right_fit):
    #行数
    y_max = img.shape[0]
    #设置填充之后的图像的大小 取到0-255之间
    out_img = np.dstack((img,img,img))*255
    #根据拟合结果，获取拟合曲线的车道线像素位置
    left_points = [[left_fit[0] * y ** 2 + left_fit[1] * y + left_fit[2], y] for y in range(y_max)]
    right_points = [[right_fit[0] * y ** 2 + right_fit[1] * y + right_fit[2], y] for y in range(y_max - 1, -1, -1)]
    # 将左右车道的像素点进行合并
    line_points = np.vstack((left_points, right_points))
    # 根据左右车道线的像素位置绘制多边形
    cv2.fillPoly(out_img, np.int_([line_points]), (0, 255, 0))
    return out_img



if __name__ == "__main__":
    #透视变换
    #获取原图的四个点
    img = cv2.imread('./test/straight_lines2.jpg')
    points = [[601, 448], [683, 448], [230, 717], [1097, 717]]
    #将四个点绘制到图像上 (文件，坐标起点，坐标终点，颜色，连接起来)
    img = cv2.line(img, (601, 448), (683, 448), (0, 0, 255), 3)
    img = cv2.line(img, (683, 448), (1097, 717), (0, 0, 255), 3)
    img = cv2.line(img, (1097, 717), (230, 717), (0, 0, 255), 3)
    img = cv2.line(img, (230, 717), (601, 448), (0, 0, 255), 3)

    #透视变换的矩阵
    M,M_inverse = cal_perspective_params(img,points)

    #车道线检测演示
    ret, mtx, dist, rvecs, tvecs = cal_calibrate_params(file_paths)
    undistort_img = img_undistort(img,mtx,dist)
    #提取车道线
    pipeline_img = pipeline(undistort_img)
    #透视变换
    trasform_img = img_perspect_transform(pipeline_img,M)
    #计算车道线的拟合结果
    left_fit,right_fit = cal_line_param(trasform_img)
    #进行填充
    result = fill_lane_poly(trasform_img,left_fit,right_fit)
    plt.figure()
    # 反转CV2中BGR 转化为matplotlib的RGB
    plt.imshow(result[:, :, ::-1])
    plt.title("vertical view:FULL")
    plt.show()

    #透视变换的逆变换
    trasform_img_inv = img_perspect_transform(result,M_inverse)
    plt.figure()
    # 反转CV2中BGR 转化为matplotlib的RGB
    plt.imshow(trasform_img_inv[:, :, ::-1])
    plt.title("Original drawing:FULL")
    plt.show()

    #与原图进行叠加
    res = cv2.addWeighted(img,1,trasform_img_inv,0.5,0)
    plt.figure()
    # 反转CV2中BGR 转化为matplotlib的RGB
    plt.imshow(res[:, :, ::-1])
    plt.title("safe work")
    plt.show()

output: