OpenCV study notes--Getting started with lane line detection

The gitee source code of this note:
https://gitee.com/hongtao-jiang/opencv_lanedetect.git

2023.8.5

1. OpenCV installation

Whether conda manages the virtual environment depends on yourself
pip install opencv

import cv2
print(cv2.__version__)

Check if the library is installed successfully

2. Reading and saving pictures

import cv2

img=cv2.imread('img.jpg',cv2.IMREAD_GRAYSCALE) #此为以灰度图格式读入，彩色读入则shape是3通道
print(type(img)) #<class 'numpy.ndarray'>
print(img.shape) #(368, 640)

cv2.imshow('image',img)#一闪而逝
#cv2.waitKey(0)#延时阻塞 0一直阻直到键入  单位毫秒
#k=cv2.waitKey(0)
#print(k)  #输出键入的ASCII值

cv2.imwrite('img_gray.jpg',img)#灰度图重组到文件里，图片要指定后缀，cv以此确定压缩格式

3. Canny algorithm edge detection

Calculate the surrounding gradient of each pixel, and compare the changes to determine whether it is an edge.
For points B and C, which are not on the edge, there is no obvious change in the gradient, but for A on the edge, the gradient changes will be large, so it is possible It is an edge point
. The above is an ideal example. The gradient is

calculated along the edge normal. In fact, the situation is more complicated. For example: 1 is the left and right edges, and 3 is both up and down. Gradient direction (there are 4 positive and negative),
(1) Apply a Gaussian filter to smooth the image and filter out noise.
Edge detection is susceptible to noise, so smooth the image and reduce noise points
(2) Calculate the gradient size and direction of each pixel of the image.


(3) Non-maximum value suppression to eliminate the adverse effects of edge detection
Traverse all pixels in the image to determine whether the current pixel is the maximum value with the same direction gradient among the surrounding pixels.

Keep the yellow background pixels, and others return to 0

(4) Apply double-threshold detection to determine the real and potential edges.

First set the high and low thresholds (generally the high threshold is 2 to 3 times the low threshold), and traverse the entire gray matrix. If the gradient of a certain point is higher than the high threshold , then set 1 in the result. If the gradient value of this point is lower than the low threshold, then set 0 in the result. If the gradient value of this point is between the high and low thresholds, the following judgment needs to be made: check the point (set It is regarded as the 8 neighbor points of the center point), and see if there is a point whose gradient value is higher than the high threshold. If it exists, it means that the center point is connected to the determined edge point, so set 1 in the result, otherwise set 0 .

(5) Suppress isolated weak edges to complete edge detection

Dual-threshold detection identifies true and potential edges

Original image:
Edge contour:

Threshold: edge_img=cv2.Canny(img,50,100)

Threshold: edge_img=cv2.Canny(img,70,120)
Compared with the previous picture, the edge line is less

import cv2

img=cv2.imread('img.jpg',cv2.IMREAD_GRAYSCALE)

edge_img=cv2.Canny(img,70,120)

cv2.imshow('edges',edge_img)
cv2.waitKey(0)
cv2.imwrite('edge_img.jpg',edge_img)

Attached is a real-time detection effect code that can dynamically adjust the threshold:

import cv2

cv2.namedWindow('edge_detection')
cv2.createTrackbar('minThreshold','edge_detection',50,1000,lambda x: x)
cv2.createTrackbar('maxThreshold','edge_detection',100,1000,lambda x: x)

img=cv2.imread('img.jpg',cv2.IMREAD_GRAYSCALE)
while True:
    minThreshold=cv2.getTrackbarPos('minThreshold', 'edge_detection')
    maxThreshold = cv2.getTrackbarPos('maxThreshold', 'edge_detection')
    edges = cv2.Canny(img, minThreshold, maxThreshold)
    cv2.imshow('edge_detection', edges)
    cv2.waitKey(10)

The effect is as follows:

used to select the appropriate threshold

4、ROI mask

region of interest area of ​​interest

· Array slicing
· Boolean operation (AND operation)

The original image is stored in the memory in the form of matrix np.array.
zeros_like generates a matrix
fillpoly of all zeros of the same size to fill the mask part, which is reserved for AND operation

cv2.fillPoly(mask,np.array([[[0,368],[240,210],[300,210],[640,368]]]),color=255)

This part is a trapezoidal fill that encloses four vertices, 255 is the gray value, pure white

import cv2
import numpy as np

edge_img=cv2.imread('edge_img.jpg',cv2.IMREAD_GRAYSCALE)

mask=np.zeros_like(edge_img)
cv2.fillPoly(mask,np.array([[[0,368],[240,210],[300,210],[640,368]]]),color=255)
cv2.imshow('mask',mask)
cv2.waitKey(0)

import cv2
import numpy as np

edge_img=cv2.imread('edge_img.jpg',cv2.IMREAD_GRAYSCALE)

mask=np.zeros_like(edge_img)
mask=cv2.fillPoly(mask,np.array([[[0,368],[240,210],[300,210],[640,368]]]),color=255)
#cv2.imshow('mask',mask)
#cv2.waitKey(0)
mask_edge_img=cv2.bitwise_and(edge_img,mask)
cv2.imshow('mask_edge_img.jpg',mask_edge_img)
#cv2.imwrite('mask_edge_img.jpg',mask_edge_img)
cv2.waitKey(0)

A bit biased, more noise, move to the right

np.array([[[0, 368], [300, 210], [340, 210], [640, 368]]]),
                    color=255)

5. Hough transform

To find a straight line, use

a line in the Cartesian coordinate system, which is a point in the Hough space;
a line in the Hough space represents all the straight lines passing through a certain point in the Cartesian system;

! ! !
Therefore, after such a corresponding transformation idea, a point in the Hough space can be used as a parameter to determine a straight line determined by most points in the Cartesian space. The right figure


is the intersection diagram of the polar coordinate system corresponding to the Hough space

As shown in the figure, two straight lines and two points in the right picture indicate that there are many intersections. According to the coordinate parameters of these two points, determine the straight line

api:
The return value is a list, which represents the coordinates of the start point and end point of the line segment.

In opencv, the Hough transform must be performed in the grayscale image. When reading the image, you must pay attention to the format

Enter line by line in the console

import cv2

import numpy as np

img=cv2.imread('lines.jpg',cv2.IMREAD_GRAYSCALE)

lines = cv2.HoughLinesP(img, 1, np.pi / 180, 15, minLineLength=40,
                        maxLineGap=20)

lines
len(lines)

We intuitively feel the 2 lines, and found that there will be 32 straight lines in the output.


In fact, it is because of the original image we are dealing with. The straight line has a width and is thicker.
Looking at the parameters, there are many lines whose starting and ending coordinates are not much different, which is just a line segment.

In order to obtain two a priori, we have to classify and divide into two groups according to the slope

import cv2
import numpy as np


def calculate_slope(line):
    """
    计算线段line的斜率
    :param line: np.array([[x_1, y_1, x_2, y_2]])
    :return:
    """
    x_1, y_1, x_2, y_2 = line[0]
    return (y_2 - y_1) / (x_2 - x_1)


edge_img = cv2.imread('mask_edge_img.jpg', cv2.IMREAD_GRAYSCALE)
# 获取所有线段
lines = cv2.HoughLinesP(edge_img, 1, np.pi / 180, 15, minLineLength=40,
                        maxLineGap=20)
# 按照斜率正负分成车道线
left_lines = [line for line in lines if calculate_slope(line) > 0]
right_lines = [line for line in lines if calculate_slope(line) < 0]

Note that in opencv, the coordinate origin is in the upper left corner

6. Outlier filtering

The previous section is divided into two groups of lines according to the slope. However, due to errors and noises, they will be misidentified as lane lines. We need to further remove some of them. We know that the lines that fall on the lane line account for the majority, and the slope difference is large. we filter out

import cv2
import numpy as np

def calculate_slope(line):
    """
    计算线段line的斜率
    :param line: np.array([[x_1, y_1, x_2, y_2]])
    :return:
    """
    x_1, y_1, x_2, y_2 = line[0]
    return (y_2 - y_1) / (x_2 - x_1)

edge_img = cv2.imread('mask_edge_img.jpg', cv2.IMREAD_GRAYSCALE)
# 获取所有线段
lines = cv2.HoughLinesP(edge_img, 1, np.pi / 180, 15, minLineLength=40, maxLineGap=20)
# 按照斜率分成车道线
left_lines = [line for line in lines if calculate_slope(line) > 0]
right_lines = [line for line in lines if calculate_slope(line) < 0]

def reject_abnormal_lines(lines, threshold):
    """
    剔除斜率不一致的线段
    :param lines: 线段集合, [np.array([[x_1, y_1, x_2, y_2]]),np.array([[x_1, y_1, x_2, y_2]]),...,np.array([[x_1, y_1, x_2, y_2]])]
    """
    slopes = [calculate_slope(line) for line in lines]#建个斜率列表
    while len(lines) > 0:
        mean = np.mean(slopes)#计算斜率平均值
        diff = [abs(s - mean) for s in slopes]#计算每个斜率和平均值的差值
        idx = np.argmax(diff)# 获取array中数值最大的 找差值最大线段的索引
        if diff[idx] > threshold:#和阈值比一下，差得多不，超标就滚蛋
            slopes.pop(idx)#斜率列表中删掉
            lines.pop(idx)#从线段列表中删掉
        else:
            break
    return lines


print('before filter:')
print('left lines number=')
print(len(left_lines))
print('right lines number=')
print(len(right_lines))

reject_abnormal_lines(left_lines, threshold=0.2)
reject_abnormal_lines(right_lines, threshold=0.2)


print('after filter:')
print('left lines number=')
print(len(left_lines))
print('right lines number=')
print(len(right_lines))

7. Least squares fitting

Fit all lines considered to be left or right lanes into a single

np.ravel: Pull high-dimensional arrays into one-dimensional arrays
np.polyfit: polynomial fitting

np.polyval: polynomial evaluation, the first parameter is the polynomial coefficient, the second parameter is any X

def least_squares_fit(lines):
    """
    将lines中的线段拟合成一条线段
    :param lines: 线段集合, [np.array([[x_1, y_1, x_2, y_2]]),np.array([[x_1, y_1, x_2, y_2]]),...,np.array([[x_1, y_1, x_2, y_2]])]
    :return: 线段上的两点,np.array([[xmin, ymin], [xmax, ymax]])
    """
    # 1. 取出所有坐标点
    x_coords = np.ravel([[line[0][0], line[0][2]] for line in lines])
    y_coords = np.ravel([[line[0][1], line[0][3]] for line in lines])
    # 2. 进行直线拟合.得到多项式系数
    poly = np.polyfit(x_coords, y_coords, deg=1)
    # 3. 根据多项式系数,计算两个直线上的点,用于唯一确定这条直线
    point_min = (np.min(x_coords), np.polyval(poly, np.min(x_coords)))
    point_max = (np.max(x_coords), np.polyval(poly, np.max(x_coords)))
    return np.array([point_min, point_max], dtype=int)


print("left lane")
print(least_squares_fit(left_lines))
print("right lane")
print(least_squares_fit(right_lines))

Add this section of processing after the code in the previous section.

Two points are practiced as a line.

8. Draw a straight line

cv2.line
add the following code

left_line = least_squares_fit(left_lines)
right_line = least_squares_fit(right_lines)

img = cv2.imread('img.jpg', cv2.IMREAD_COLOR)
cv2.line(img, tuple(left_line[0]), tuple(left_line[1]), color=(0, 255, 255), thickness=5)
cv2.line(img, tuple(right_line[0]), tuple(right_line[1]), color=(0, 255, 255), thickness=5)

cv2.imshow('lane', img)
cv2.waitKey(0)

Why is there a large deviation of the left lane line visible to the naked eye?
Because the Canny edge detection threshold is adjusted, and the mask area at the front ROI mask is not set properly, as shown in the figure below, there is an edge of the vehicle at the left lane line, I did not carefully adjust the threshold, and the rear mask area is not covered

. Move to get a new picture and take part in the drawing:

so the adjustment depends on the situation

9. Video stream reading and writing

capture = cv2.VideoCapture('video.mp4')
while True:
    ret, frame = capture.read()#ret视频流状况，是否关闭
    frame = show_lane(frame)
    cv2.imshow('frame', frame)
    cv2.waitKey(100)