opencv lane line detection

This learning content is to use the opencv library function to detect lane lines in a video.

The implementation steps are divided into:

1. Camera calibration to correct distorted images

2. Perspective change to show the area of interest (lane line area)

3. Extract white lane lines through the hls image, and extract yellow lane lines through the lab image.

4. Use sliding windows to identify lane lines

5. Binomial fitting to find the lane line curve

6. Find the radius of curvature and deviation position

7. Back-project onto the original image

8. Display the detection process

The first step is to achieve camera calibration

Transfer the coordinate information of the world coordinate system to the pixel coordinate system to determine the relationship between the 3D geometric position in the space and the corresponding point in the image.

The coordinate systems involved in transformation include world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system.

The first is the world coordinate system to the camera coordinate system

The changes between the two coordinate systems as rotation and translation can be expressed by the rotation and translation matrix $\begin{pmatrix}Xc \\ Yc \\ Zc \end{pmatrix}=\begin{pmatrix}R &t \\ 0 & 1 \end{pmatrix}\cdot \begin{pmatrix}Xw \\ Yw \\ Zw \end{pmatrix}$

Where R is the rotation matrix and t is the translation matrix.

Next is the transformation from the camera coordinate system to the image coordinate system

The relationship can be derived from similar triangles $\frac{f}{Zc}=\frac{x}{Xc}=\frac{y}{Yc}$ and then the matrix form is written as $Zc\begin{pmatrix}x \\ y \\ 1 \end{pmatrix}=\begin{pmatrix}f & 0 &0 &0 \\ 0 & f & 0 &0 \\ 0 & 0 & 1 & 0 \end{pmatrix}\cdot \begin{pmatrix}Xc \\ Yc \\ Zc \\ 1 \end{pmatrix}$ f is the focal length.

Image coordinate system to pixel coordinate system

$\begin{pmatrix}u \\ v \\ 1 \end{pmatrix}=\begin{pmatrix}\frac{x}{dx} &0 &u0 \\ 0 &\frac{y}{dy} &v0 \\ 0 & 0 & 1 \end{pmatrix}\cdot \begin{pmatrix}x \\ y \\ 1 \end{pmatrix}$

In summary, the changes are: $Zc\begin{pmatrix}u \\ v \\ 1 \end{pmatrix}=\begin{pmatrix}\frac{x}{dx} &0 &u0 \\ 0 &\frac{y}{dy} &v0 \\ 0 & 0 & 1 \end{pmatrix}\cdot\begin{pmatrix}f & 0 &0 &0 \\ 0 & f & 0 &0 \\ 0 & 0 & 1 & 0 \end{pmatrix}\cdot \begin{pmatrix}R &t \\ 0 & 1 \end{pmatrix}\cdot \begin{pmatrix}Xw \\ Yw \\ Zw \end{pmatrix}$

Due to the influence of the lens, the image will be distorted (mainly divided into two categories: tangential and radial distortion)

Tangential: $x_{cor}=x+2p_{1}xy+p_{2}(r^{2}+2x^{2}) y_{cor}=y+2p_{2}xy+p_{1}(r^{2}+2y^{2})$

Radial: $x_{cor}=x(1+k_{1}r^{2}+k_{2}r^{4}+k_{3}r^{6}) y_{cor}=y(1+k_{1}r^{2}+k_{2}r^{4}+k_{3}r^{6})$

The next step is to transfer the corrected points to the pixel plane through internal parameters.

$\left\{\begin{matrix}u=f_{x}x_{cor}+u_{0} \\ v=f_{y}y_{cor}+v_{0} \end{matrix}\right.$ where fx is x/dx, fy is the same

This article implements camera calibration and distortion correction through the opencv function

The camera calibration is completed through checkerboard shots taken at different angles. The checkerboard data set in this article is as shown in the figure below

paths=glob.glob(r'../IR_camera_calib_img/ca*.jpg')

The 20 chessboard positions are obtained by the glob.glob() function, and the function return value is stored in list form.

ret,corners=cv2.findChessboardCorners(gray,(9,6),None)

Next, use cv2.findChessboardCorners to find the pixel coordinates of the internal intersection points of the checkerboard. From the checkerboard picture above, we can see that the internal corner points are w=9 and h=6.

Since the world coordinates can be set by themselves, this article sets the Z axis of the world coordinate system to 0. From this, the world coordinate system is obtained in the form of (1,0,0), (2,0,0), (3,0,0)... The world coordinates of the internal intersection points are saved with wld_object, and through cv2. The intersection pixel coordinate system obtained by findChessboardCorners is saved by img_object=[]. The next step is to bring in cv2.calibrateCamera() to obtain mtx, which is the internal parameter matrix of the camera; the camera distortion parameter expressed by dist; rvecs represents the rotation parameter:; tvecs represents translation vectors, is the translation parameter.

Finally, distortion correction is completed through cv2.undistort(img1,mtx,dis,None,mtx)

The implementation code is as follows

worldobject=np.zeros((9*6,3),np.float32)
worldobject[:,:2]=np.mgrid[0:9,0:6].T.reshape(-1,2)

#
wld_object=[]
img_object=[]
paths=glob.glob(r'../IR_camera_calib_img/ca*.jpg')

#plt.figure(figsize=(10, 6))
#print(paths)
for path in paths:
    img= cv2.imread(path,flags=1)
    gray=cv2.cvtColor(img,cv2.COLOR_BGR2GRAY)

    #找内角点
    ret,corners=cv2.findChessboardCorners(gray,(9,6),None)


    if ret == True:
        wld_object.append(worldobject)
        img_object.append(corners)
        i+=1
        #cv2.drawChessboardCorners(img,(11,8),corners,ret)
        #cv2.imshow('img',img)
        #plt.subplot(3,6,i)
        #plt.imshow(img)
        #cv2.waitKey(0)
        #cv2.destroyAllWindows()
#plt.show()

#img1=cv2.imread('100000.png')
ret,mtx,dis,rvecs,tvecs=cv2.calibrateCamera(wld_object,img_object,gray.shape[::-1],None,None)


#校正
img1=cv2.imread('straight_lines1.jpg',)
cv2.imshow('o',img1)
dst=cv2.undistort(img1,mtx,dis,None,mtx)
cv2.imshow('d',dst)
cv2.waitKey(0)
cv2.imwrite('correct.jpg',dst)
cv2.destroyAllWindows()

achieve results

The second step is to change the perspective to obtain the area of interest.

In the original image, the lane line area can be considered a trapezoid. Now we need to turn this area into the scene we see when looking down to facilitate the subsequent recognition operation of the lane line. Select this article

scr=np.float32([[575,460],[700,460],[1096,720],[200,720]])
dst=np.float32([[200,0],[950,0],[950,720],[200,720]])

scr can be considered as the trapezoidal area in the original image. dst can be considered as the rectangular area where we look down at the lane line and obtain the transformation matrix through cv2.getPerspectiveTransform().

Then bring in cv2.warpPerspective(img,M,(w,h),flags=cv2.INTER_LINEAR) to complete the perspective transformation.

import cv2
import numpy as np
import matplotlib.pyplot as plt

img=cv2.imread('correct.jpg')
h,w=img.shape[:2]
scr=np.float32([[575,460],[700,460],[1096,720],[200,720]])
dst=np.float32([[200,0],[950,0],[950,720],[200,720]])
M=cv2.getPerspectiveTransform(scr,dst,None)
Minv=cv2.getPerspectiveTransform(dst,scr,None)
img1=cv2.warpPerspective(img,M,(w,h),flags=cv2.INTER_LINEAR)
plt.figure(figsize=(10,6))
plt.subplot(121),plt.imshow(img)
plt.subplot(122),plt.imshow(img1)
plt.show()

Next, the third step begins to identify the lane lines. Since the lane lines are yellow and white, the purpose now is to separate the yellow lines and white lines and finally combine them on one picture to complete the detection.

The first is to extract the white car lines through the HSL color space

hls=cv2.cvtColor(img1,cv2.COLOR_BGR2HLS)
lab=cv2.cvtColor(img1,cv2.COLOR_BGR2Lab)

#print(hls)
L_channel=hls[:,:,1]
L_channel=L_channel*(255/np.max(L_channel))
h,w=L_channel.shape[:2]

binary_out=np.zeros_like(L_channel)
for i in range(h):
    for j in range(w):
        if L_channel[i][j]>=220 and L_channel[i][j]<=255:
            binary_out[i][j]=1

Then extract the yellow lane lines through Lab color space

b_channel=lab[:,:,2]
if b_channel.max()>100:
    b_channel=b_channel*(255/np.max(b_channel))
lab_binary=np.zeros_like(b_channel)
for i in range(h):
    for j in range(w):
        if b_channel[i][j]>=195 and b_channel[i][j]<=255:
            lab_binary[i][j]=1

out=np.zeros_like(lab_binary)
for i in range(h):
    for j in range(w):
        if binary_out[i][j]==1 or lab_binary[i][j]==1:
            out[i][j]=1

Next is the sliding window recognition of lane lines

After extracting the lane lines, you can obtain a binary image that is either black or white. You can use histogram statistics to determine the position of the lane lines on the image. As shown in the histogram below, the horizontal axis is the horizontal coordinate system of the image, and the vertical axis is is the number of pixels

Implementation code

Histogram statistics are obtained by np.sum() axis=0 is compressed rows, and each column is added

histogram = np.sum(out[:,:], axis=0)
x=np.arange(1280)
plt.plot(x,histogram, linewidth=2)

The next step is to implement the sliding window. First define the window size. This experiment uses 9 windows, so the height is img.shape[0]/9

Then find the maximum value of the lane line in the image (use np.argmax), leftx_base, and rightx_base are the midpoints of the first window respectively. Set the width to 100 to get a window with a width of 200 and a height of 80.

Next, start traversing the image to find non-zero pixels, and save its index value in left_lane_inds (left lane line) through nonzero. The same goes for the right lane line.

During the search process, if the non-zero pixel points exceed a certain threshold, the coordinates of the window center point will be changed.

The above is the implementation of sliding window

leftx_base = np.argmax(histogram[:midpoint])
rightx_base = np.argmax(histogram[midpoint:]) + midpoint
#窗口
windows=9
windows_height=out.shape[0]//windows
margin=100
minpix=50
#统计
nonzero=out.nonzero()
nonzeroy=np.array(nonzero[0])
nonzerox=np.array(nonzero[1])

Left_lane_inds=[]
Right_lane_inds=[]
leftx_current = leftx_base
rightx_current = rightx_base

for window in range(windows):
    win_y_low=out.shape[0]-(window+1)*windows_height
    win_y_high = out.shape[0] - window  * windows_height
    win_xleft_low= leftx_current-margin
    win_xleft_high = leftx_current + margin
    win_xright_low = rightx_current - margin
    win_xright_high = rightx_current + margin

    #Draw
    cv2.rectangle(out_img,(win_xleft_low,win_y_low),(win_xleft_high,win_y_high),[0,255,0],2)
    cv2.rectangle(out_img, (win_xright_low, win_y_low), (win_xright_high, win_y_high), [0, 255, 0], 2)

    good_left_inds =((nonzerox>=win_xleft_low)&(nonzerox<win_xleft_high)&(nonzeroy>=win_y_low)&(nonzeroy<win_y_high)).nonzero()[0]
    good_right_inds =((nonzerox>=win_xright_low)&(nonzerox<win_xright_high)&(nonzeroy>=win_y_low)&(nonzeroy<win_y_high)).nonzero()[0]

    Left_lane_inds.append(good_left_inds)
    Right_lane_inds.append(good_right_inds)

    if len(good_left_inds)>minpix:leftx_current = np.int32(np.mean(nonzerox[good_left_inds]))
    if len(good_right_inds) > minpix: rightx_current = np.int32(np.mean(nonzerox[good_right_inds]))

Left_lane_inds=np.concatenate(Left_lane_inds)
Right_lane_inds=np.concatenate(Right_lane_inds)

lefttx=nonzerox[Left_lane_inds]
leftty=nonzeroy[Left_lane_inds]
righttx=nonzerox[Right_lane_inds]
rightty=nonzeroy[Right_lane_inds]


left_fit=np.polyfit(leftty,lefttx,2)
right_fit=np.polyfit(rightty,righttx,2)

ploty=np.linspace(0,out.shape[0]-1,out.shape[0])
left_fitx=left_fit[0]*ploty**2+ploty*left_fit[1]+left_fit[2]
right_fitx=right_fit[0]*ploty**2+ploty*right_fit[1]+right_fit[2]
out_img[leftty, lefttx] = [255, 0, 0]
out_img[rightty, righttx] = [0, 0, 255]

#plt.plot(left_fitx, ploty,  linewidth=2)
#plt.plot(right_fitx, ploty,  linewidth=2)

plt.subplot(121),plt.imshow(img1,'gray')
plt.subplot(122),plt.imshow(out_img,'gray')
plt.show()

Then use the index value of the non-0 point obtained in the previous step (that is, the horizontal and vertical coordinates of the pixel) to fit the binomial polynomial to get the following picture

The quadratic function is expressed as y=ax**2+bx+c. Use the function np.polyfit() to bring in the coordinate values and return [a, b, c] three coefficients of the quadratic function, and ploty can be considered to represent the natural Variables are obtained through np.linspace

Through the quadratic term coefficients obtained above, the coordinates (x, y) of the quadratic term curve are obtained in the image and saved with pts_left and pts_right respectively. The colors between the lane lines are filled with cv2.fillpoly and finally back-projected onto the original image.

 ploty=np.linspace(0,binary_warped.shape[0]-1,binary_warped.shape[0])
    left_fitx = left_fit[0] * ploty ** 2 + left_fit[1] * ploty + left_fit[2]
    right_fitx = right_fit[0] * ploty ** 2 + right_fit[1] * ploty + right_fit[2]
    warp_zero = np.zeros_like(binary_warped).astype(np.uint8)
    color_warp=np.dstack((warp_zero,warp_zero,warp_zero))
    pts_left = np.array([np.transpose(np.vstack([left_fitx, ploty]))])
    pts_right = np.array([np.flipud(np.transpose(np.vstack([right_fitx, ploty])))])
    pts = np.hstack((pts_left, pts_right))
    cv2.fillPoly(color_warp,np.int_([pts]),(0,255,0))

    newwarp=cv2.warpPerspective(color_warp, Minv, (undist.shape[1], undist.shape[0]))
    result = cv2.addWeighted(undist, 1, newwarp, 0.3, 0)

There are also calculations of the radius of curvature and deviation position.

Since it is a quadratic function y=ax**2+bx+c, the radius of curvature can be obtained by the formula: $\frac{(1+{y}'^{2})^{1.5}}{{y}''}$

The deviation position is the difference between the distance between lane lines and the distance from the center of the image.

    ploty = np.linspace(0, binary_warped.shape[0] - 1, binary_warped.shape[0])
    leftx = left_fit[0] * ploty ** 2 + left_fit[1] * ploty + left_fit[2]
    rightx = right_fit[0] * ploty ** 2 + right_fit[1] * ploty + right_fit[2]

    ym_per_pix = 30 / 720  # meters per pixel in y dimension
    xm_per_pix = 3.7 / 700  # meters per pixel in x dimension
    y_eval = np.max(ploty)
    left_cur_fit=np.polyfit(ploty*ym_per_pix,leftx*xm_per_pix,2)
    right_cur_fit=np.polyfit(ploty*ym_per_pix,rightx*xm_per_pix,2)

    left_curverad=((1 + (2*left_cur_fit[0]*y_eval*ym_per_pix + left_cur_fit[1])**2)**1.5) / np.absolute(2*left_cur_fit[0])
    right_curverad=((1 + (2*right_cur_fit[0]*y_eval*ym_per_pix + right_cur_fit[1])**2)**1.5) / np.absolute(2*right_cur_fit[0])
    curverad=(left_curverad+right_curverad)/2.
    #
    lane_width=np.absolute(leftx[719]-rightx[719])
    lane_xm_per_pix=3.7/lane_width
    veh_pos=(leftx[719]+rightx[719])*lane_xm_per_pix/2.
    cen_pos=binary_warped.shape[1]*lane_xm_per_pix/2.
    distance_from_center=np.absolute(cen_pos-veh_pos)

Final video

Lane line detection

opencv lane line detection

Guess you like