The whole process of license plate recognition project - self-study of opencv knowledge

what is opencv?

1. OpenCV (Open Source Computer Vision Library) is an open source computer vision and machine learning library that provides interfaces such as C++, C, and python, and supports Windows, Linux, Android, and MacOS platforms.
2. After 2016, the application of deep learning has become more and more extensive, and modules such as CNN have also been added to OpenCV, which can be connected with the models trained by Tensorflow and Caffe2.

Image Digitization Fundamentals

A picture is made up of countless small squares.
Color pictures - are composed of small colored squares, each colored square is quantized by a vector of three values . (0, 0, 0) represents black, (255, 255, 255) represents white, and (255, 0, 0) represents red.
Gray image - is composed of small gray squares, each of which is quantized by a vector of numerical values .

A two-dimensional numeric matrix with 508 rows (height) and 672 columns (width).

The computer will digitize each pixel into a numerical value. The "bit depth" of the grayscale image is 8 bits (color is 24 bits), which means that each square is digitized into a uchar type number between [0, 255], which is used 256 numbers are used to measure the shade of gray, the larger the value, the brighter the value, the smaller the value, the more gray, 255 is white, and 0 is black.

image acquisition

import cv2
gray_img = cv2.imread(r'C:\Users\fujie\Pictures\shunli.jpg',1) # 读入图像
cv2.imshow('Image', gray_img) # 显示图像
cv2.imwrite('images/output.jpg', gray_img)     # 保存图像
cv2.waitKey()  #等待时间，毫秒级，0表示任意中止

image transformation

Performing some operations on images can be regarded as a means of image expansion .

• The purpose of image enhancement is to improve the visual effect of the image. For the application of a given image, purposefully enhance the overall or local characteristics of the image, make the original unclear image clear or enhance some interesting features, expand the image. The difference between the features of different objects in the image, suppress the uninteresting features, improve the image quality, enrich the amount of information, interpret and recognize the strong image, and meet the needs of some feature analysis.

change size

(x,y,3)
row x - height
column y - width

License plate recognition content analysis

1. The automatic recognition technology of license plate is to connect the method of processing images with the software technology of the computer , in order to accurately recognize the characters of the license plate, and transmit the recognized data to the traffic real-time management system, so as to finally realize the traffic supervision. Features.
2. my country's car license plate is generally composed of seven characters and a dot.
   License plate recognition is roughly divided into four parts: image acquisition (obtaining the image information of the vehicle from the visual sensor) - locating the license plate (finding the position of the license plate from the original image) - character segmentation (segmenting the license plate into seven characters) Used for subsequent character recognition) - character recognition (recognizing the separated single characters to identify the value of the license plate)

Locate the license plate

Find the location of the license plate (locate the area) from the acquired image, and segment the license plate from this area for subsequent character segmentation.
There are four main types of license plate positioning methods:

1. Color-based segmentation: Utilize color space information, including color edge algorithm, color distance algorithm, and similarity algorithm.
2. Texture-based segmentation: For example, using the texture features in the horizontal direction of the license plate area to perform segmentation, including wavelet texture and horizontal gradient difference texture.
3. Segmentation based on edge detection
4. Segmentation Based on Mathematical Morphology

Image noise reduction

To acquire an image, the original image is color and becomes a grayscale image for subsequent processing.

The original image contains noise information. Noise can be understood as random changes in grayscale values ​​caused by one or more reasons. In most cases, smoothing techniques (filtering) are required to remove them. Commonly used smoothing algorithms include Gaussian smoothing and mean smoothing based on two-dimensional discrete convolution. . . This operation uses the Gaussian smoothing method.

Gaussian filtering : If the value of a pixel is much higher than the surrounding points, it may be a noise or high-frequency edge. Gaussian filtering uses multiple points around the point to make a weighted average of them, which is equivalent to using the surrounding value to lower this high. value, which is called smoothing. The mathematical embodiment of Gaussian filtering is the operation of reassigning the pixel values ​​of the entire image through weighted averaging. (The weighted average can be understood as the result of calculating and adding different parts according to different proportions.)
Details link

- The resulting image is slightly blurred compared to the original. In this way, a single pixel becomes almost insignificant on the Gaussian smoothed image.

• Gaussian smoothing of OpenCV:
  Gaussian smoothing is done by doing a two-dimensional discrete convolution operation between two matrices, and the Gaussian convolution kernel is convolved with the original image.
  

• convolution:
  
• The function to build a Gaussian convolution kernel in one-dimensional vertical direction is given in OpenCV: Mat getGaussianKernel(int ksize, double sigma, in ktype = CV/_64F)

Morphological processing

Morphological processing is mainly used to extract image components that are meaningful for expressing and describing the shape of the region from the image, so that the subsequent recognition work can capture the most essential (most discriminative) shape features of the target object, such as boundaries and shapes. connected areas, etc. At the same time, techniques such as thinning, pixelation, and burr trimming are often used in image preprocessing and postprocessing, becoming a powerful complement to image enhancement techniques.

Basic operations of morphological processing: erosion, expansion, opening operation, closing operation, hit and miss, skeleton extraction, etc. For example, erosion makes dark areas larger, and dilation makes bright areas larger.

Perform the opening operation ( the opening operation can make the outline of the image smooth, and can also disconnect the narrow connections and eliminate the fine burrs . This operation pave the way for more accurate extraction of the outline of the license plate later):

Open operation details link
Open operation is mainly black and white, and the black area becomes larger. Remove useless information points.
The closed operation is mainly to eat black for nothing, and the white area becomes larger. Increase the white highlight area.

Combine the original grayscale image and the morphological image into one image for subsequent threshold segmentation operations:

Threshold segmentation + edge detection

Threshold segmentation is a common algorithm for segmenting images directly, and it depends on the gray value of image pixels. Corresponding to a single target image, only one threshold can be selected to divide the image into two categories: target and background. This is called single-threshold segmentation; if the target image is complex, multiple thresholds can be selected to separate the target area and background in the image. It is divided into multiple pieces, which is called multi-threshold segmentation. At this time, it is also necessary to distinguish the image targets in the detection results, and uniquely identify each image target area to distinguish.

By thresholding, we hope to be able to separate our research object from the background. The image is divided into several specific regions with unique properties, each region represents a collection of pixels, and each collection represents an object.

Using Ostu threshold processing : When using the function cv.threshold() for threshold processing, you need to customize a threshold, and use this threshold as the basis for image threshold processing. OTSU can help us traverse all possible thresholds to find the best one.

Do closed-run smoothing (no edge detection after threshold segmentation)

Add edge detection after threshold segmentation: (use canny edge detection here)

and then perform a closing operation to increase the white highlight area:

perform an opening operation to remove the useless white part and fill the gap:

Finally, perform expansion to fill the gap again and increase the white highlight part:

Each contour can be regarded as an ordered set of points (pixels), and now we want to extract the contours of these white areas. (draw the original picture with the outline)

Now that we have the outline, we need to filter out the outline where the license plate is located (you can see that the area where the license plate is located is a rectangle), since the ratio of the license plate width and height is fixed (the license plate The height and width of the characters are fixed, 90mm and 45mm respectively), according to this geometric feature, we filter, and then use the green lines to select the obtained license plate frame, and at the same time intercept the license plate for the next character segmentation. .

Locate the full code of the license plate:

import cv2
import numpy as np
import matplotlib.pyplot as plt
from scipy import signal
#########该函数能够读取磁盘中的图片文件，默认以彩色图像的方式进行读取
def imread_photo(filename, flags=cv2.IMREAD_COLOR):
    return cv2.imread(filename, flags)


##############这个函数的作用就是来调整图像的尺寸大小，当输入图像尺寸的宽度大于阈值（默认1000），我们会将图像按比例缩小#######
def resize_photo(imgArr,MAX_WIDTH = 1000):
    img = imgArr
    rows, cols= img.shape[:2]     #获取输入图像的高和宽即第0列和第1列
    if cols >  MAX_WIDTH:
        change_rate = MAX_WIDTH / cols
        img = cv2.resize(img ,( MAX_WIDTH ,int(rows * change_rate) ), interpolation = cv2.INTER_AREA)
    return img

# ################高斯平滑###############
#我们首先会对图像水平方向进行卷积，然后再对垂直方向进行卷积，其中sigma代表高斯卷积核的标准差
def gaussBlur(image,sigma,H,W,_boundary = 'fill', _fillvalue = 0):
    #水平方向上的高斯卷积核
    gaussKenrnel_x = cv2.getGaussianKernel(sigma,W,cv2.CV_64F)
    #进行转置
    gaussKenrnel_x = np.transpose(gaussKenrnel_x)
    #图像矩阵与水平高斯核卷积
    gaussBlur_x = signal.convolve2d(image,gaussKenrnel_x,mode='same',boundary=_boundary,fillvalue=_fillvalue)
    #构建垂直方向上的卷积核
    gaussKenrnel_y = cv2.getGaussianKernel(sigma,H,cv2.CV_64F)
    #图像与垂直方向上的高斯核卷积核
    gaussBlur_xy = signal.convolve2d(gaussBlur_x,gaussKenrnel_y,mode='same',boundary= _boundary,fillvalue=_fillvalue)
    return gaussBlur_xy


def chose_licence_plate(contours, Min_Area=2000):
    temp_contours = []
    for contour in contours:
        if cv2.contourArea(contour) > Min_Area:
            temp_contours.append(contour)
    car_plate = []
    for temp_contour in temp_contours:
        rect_tupple = cv2.minAreaRect(temp_contour)
        rect_width, rect_height = rect_tupple[1]
        if rect_width < rect_height:
            rect_width, rect_height = rect_height, rect_width
        aspect_ratio = rect_width / rect_height
        # 车牌正常情况下宽高比在2 - 5.5之间
        if aspect_ratio > 2 and aspect_ratio < 5.5:
            car_plate.append(temp_contour)
            rect_vertices = cv2.boxPoints(rect_tupple)
            rect_vertices = np.int0(rect_vertices)
    return car_plate

def license_segment( car_plates ):
    if len(car_plates)==1:
        for car_plate in car_plates:
            row_min,col_min = np.min(car_plate[:,0,:],axis=0)
            row_max, col_max = np.max(car_plate[:, 0, :], axis=0)
            cv2.rectangle(img, (row_min,col_min), (row_max, col_max), (0,255,0), 2)
            card_img = img[col_min:col_max,row_min:row_max,:]
            cv2.imshow("img", img)
        cv2.imwrite( "card_img.png", card_img)
        cv2.imshow("card_img.png", card_img)
        cv2.waitKey(0)
        cv2.destroyAllWindows()
    return  "card_img.png"


if __name__ == "__main__":

    img = imread_photo("car_test.png")  # 默认读取彩色图片
    gray_img = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)  # 将彩色图片转换为灰色图片,灰色图像更便于后续处理。
    rgray_img = resize_photo(gray_img)

    # 高斯平滑
    blurImage = gaussBlur(gray_img, 5, 400, 400, 'symm')
    #对bIurImage进行灰度级显示
    blurImage = np.round(blurImage)
    blurImage = blurImage.astype(np.uint8)

    kernel = np.ones((10, 10), np.uint8)
    #开运算
    img_opening = cv2.morphologyEx(blurImage, cv2.MORPH_OPEN, kernel)
    # cv2.imshow("GaussBlur_gray_img", blurImage)
    # cv2.imshow("xingtai_gray_img", img_opening)


    #将两幅图像合成为一幅图像
    img_opening = cv2.addWeighted(rgray_img, 1, img_opening, -1, 0)
    # cv2.imshow("hecheng_gray_img", img_opening)



    #阈值分割
    t, result_img = cv2.threshold(img_opening, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
    # cv2.imshow("yuzhi_gray_img",result_img)
    #canny边缘检测
    img_edge = cv2.Canny(result_img, 100, 200)
    # cv2.imshow("bianyuan_gray_img", img_edge)
    #闭运算来填充白色物体内细小黑色空洞的区域并平滑其边界
    kernel1 = np.ones((18, 18), np.uint8)
    img_edge1 = cv2.morphologyEx(img_edge, cv2.MORPH_CLOSE, kernel1)
    # cv2.imshow("biyunsuan", img_edge1)

    kernel2 = np.ones((10, 10), np.uint8)
    img_edge2 = cv2.morphologyEx(img_edge1, cv2.MORPH_OPEN, kernel2)
    # cv2.imshow("kaiyunsuan", img_edge2)

    kernel = np.ones((20, 20), np.uint8)
    img_dilate = cv2.dilate(img_edge2, kernel)  # 膨胀
    cv2.imshow("dilate", img_dilate)  # 显示图片

    # #查找图像边缘整体形成的矩形区域, contours是找到的多个轮廓
    image, contours, hierarchy = cv2.findContours(img_dilate,cv2.RETR_TREE, cv2.CHAIN_APPROX_NONE)
    # cv2.imshow("xunzhao", hierarchy)

    draw_img = img.copy()
    result = cv2.drawContours(draw_img, contours, -1, (0, 0, 255), 2)
    # 画出带有轮廓的原始图片
    cv2.imshow('ret',result)
    # cv2.waitKey(0)
    # cv2.destroyAllWindows()

    car_plates = chose_licence_plate(contours)
    card_img = license_segment(car_plates)