Tensorflow license plate recognition complete project (including complete source code and training set)

Design and implementation of license plate recognition system based on TensorFlow, using tensorflow and OpenCV related technologies to realize license plate positioning, license plate binarization, license plate denoising enhancement, image segmentation, model training and license plate recognition, etc.

Project problems, completed, Daiso can chat with bloggers privately

content

Environmental preparation

Thought process

Function description

detail

Overall framework of the project

Process display

Technical introduction

Environmental preparation

   Anaconda 4.10.3
   Tensorflow 2.6.0
   python3.7.8
   coding: utf-8
   pycharm interpreter: D:\Anaconda\envs\tensorflow\python.exe
   and various third-party libraries

Thought process

1. Recognize and locate the license plate through opencv cutting, cut and save

2. Identify the province abbreviation, identify the city code, identify the license plate number

Function description

   car_num_main.py : Convert the image to grayscale image, polarize the grayscale image, segment the image and save it as .jpg and .bmp files
   respectively train-license-province.py : Province abbreviation training recognition
   train-license-letters.py : City code training recognition
   train-license-digits.py : License plate number training recognition

Four core programs download

detail

   1. After cutting the images, save    them in two folders
   . Execute the following command
for province abbreviation training
   input: python train-license-province.py predict Execute the following command for province abbreviation recognition
   input: python train-license-letters.py train Execute the following command for city code training
   input: python train-license- letters.py predict for city code recognition
   input and execute the following command: python train-license-digits.py train for license plate number training
   input and execute the following command: python train-license-digits.py predict for license plate number recognition
   3. Pictures to be recognized Adjust it to .jpg format, and adjust the size to 600*413 pixels. It is best to debug according to the code
   . 4. For the license plate number that can be accurately recognized, please refer to the training set in the data set

test dataset

Overall framework of the project

capture_img : store the license plate image to be recognized

img_cut: After running car_num_main.py, a cut image is generated

img_cut_not_3240 : Run car_num_main.py to generate a cut image (contrast enhancement)

test_images : store test images

train_images : store training images

train-saver : train the model

Process display

PS D:\pycharm\pycharm_work> cd .\chepai\
PS D:\pycharm\pycharm_work\chepai> python train-license-province.py train

PS D:\pycharm\pycharm_work\chepai> python train-license-province.py predict

PS D:\pycharm\pycharm_work\chepai> python train-license-digits.py predict

Technical introduction

1. Tensorflow

TensorFlow is an open-source software library for high-performance numerical computing. With its flexible architecture, users can easily deploy computing jobs to multiple platforms (CPU, GPU, TPU) and devices (desktops, server clusters, mobile devices, edge devices, etc.).

TensorFlow is an open source machine learning library for research and production. TensorFlow provides a variety of APIs for beginners and experts to develop in desktop, mobile, web, and cloud environments.

TensorFlow uses data flow graphs to calculate, so first we have to create a data flow graph, and then put our data (data in the form of tensors) in the data flow graph for calculation . Nodes represent mathematical operations in the graph, and edges in the graph represent multidimensional data arrays that are interconnected between nodes, namely tensors. When training the model, the tensor will continuously flow from one node in the data flow graph to another node.

2. OpenCV

OpenCV is a cross-platform computer vision and machine learning software library released under the BSD license (open source) and can run on Linux , Windows , Android and Mac OS operating systems. It is lightweight and efficient - it consists of a series of C functions and a small number of C++ classes, and provides interfaces to languages such as Python, Ruby, and MATLAB, and implements many general algorithms in image processing and computer vision.

The visual processing algorithms provided by OpenCV are very rich, and it is partly written in C language. With its open source features, it can be processed properly without adding new external support. It can also be completely compiled and linked to generate execution programs, so many people use it to To transplant the algorithm, the OpenCV code can run normally in DSP system and ARM embedded system after appropriate rewriting. Its application fields such as: human-computer interaction, object recognition, image segmentation , face recognition , action recognition, motion tracking , robotics, motion analysis, machine vision, structural analysis, vehicle safety driving and other fields.

system design

Automatic license plate recognition is based on computer vision processing, digital image processing, pattern recognition and other technologies. It processes and analyzes vehicle images or video images captured by cameras to obtain the license plate number of each vehicle, thereby completing the recognition process. On this basis, various traffic supervision functions such as parking lot entrance and exit charge management, robbery vehicle management, highway overspeed automation management, red light electronic police, and highway toll management can be realized.

1. System processing flow

The design of automatic license plate recognition system includes vehicle image acquisition, license plate area positioning, license plate feature contour extraction and license plate content recognition.

2. License plate acquisition

License plate image acquisition is the first step in license plate recognition. The license plate image can be extracted from the vehicle image or video image captured by the camera. The license plate image can also be obtained by the user's mobile phone and then transmitted to the license plate recognition system.

3. Grayscale image generation

The vehicle image containing the license plate information captured by the camera is colored. In order to improve the anti-interference ability of the license plate recognition, the color vehicle image is firstly generated into a binary grayscale image to realize the license plate area location based on the tone. Since domestic license plates are often white on a blue background, a binary grayscale image can be generated by using the hue or color saturation characteristics of the image, so as to locate the license plate position more accurately.

Fourth, the license plate area positioning

The localization of the license plate area adopts a shape-based method. Since the background of the vehicle image is relatively complex, the initial screening should be carried out according to the characteristics of the license plate. The characteristics of the license plate can be selected from the size and proportion of the Chinese license plate, because the license plate is a fixed rectangular shape, by first finding the areas with rectangular features on the image, then extracting these areas, and then combining the length and width of the license plate. The corresponding rectangular area is obtained, so as to realize the accurate positioning of the license plate.

5. Feature contour extraction

The cv2.fifindContours() function is used in the interface between OpenCV and Python to find the contours of the detected objects.

Figure 3 and Figure 4 are the effect comparison diagrams before and after feature contour extraction:

6. License Plate Content Recognition

When the license plate content is recognized, the final license plate area is determined by calculating the maximum value of the blue value (mean value) of the candidate license plate area. For the selected license plate outline, first perform rough positioning, that is, perform left and right boundary regression processing on the license plate, remove the redundant parts on both sides of the license plate, and then perform fine positioning, that is, send the license plate to the CRNN network for character recognition, and use the left and right boundary regression model to predict The left and right borders of the license plate are drawn out, and further trimmed for precise positioning. The method based on text features is to identify according to the features of text contours, and obtain the results after corresponding algorithm analysis.

Project realization

core code display

1. Check the license plate

def find_car_num_brod():
    watch_cascade = cv2.CascadeClassifier('D:\pycharm\pycharm_work\chepai\cascade.xml')
    # 先读取图片
    image = cv2.imread("D:\pycharm\pycharm_work\chepai\capture_img\car1.jpg")
    resize_h = 1000
    height = image.shape[0]
    scale = image.shape[1] / float(image.shape[0])
    image = cv2.resize(image, (int(scale * resize_h), resize_h))
    image_gray = cv2.cvtColor(image, cv2.COLOR_RGB2GRAY)
    watches = watch_cascade.detectMultiScale(image_gray, 1.2, 2, minSize=(36, 9), maxSize=(36 * 40, 9 * 40))

    print("检测到车牌数", len(watches))
    for (x, y, w, h) in watches:
        cv2.rectangle(image, (x, y), (x + w, y + h), (0, 0, 255), 1)
        cut_img = image[y + 5:y - 5 + h, x + 8:x  + w]  # 裁剪坐标为[y0:y1, x0:x1]
        cut_gray = cv2.cvtColor(cut_img, cv2.COLOR_RGB2GRAY)

        cv2.imwrite("D:\pycharm\pycharm_work\chepai\\num_for_car.jpg", cut_gray)
        im = Image.open("D:\pycharm\pycharm_work\chepai\\num_for_car.jpg")
        size = 720, 180
        mmm = im.resize(size, Image.ANTIALIAS)
        mmm.save("D:\pycharm\pycharm_work\chepai\\num_for_car.jpg", "JPEG", quality=90)
        break

2. Binarized image

def cut_car_num_for_chart():
    # 1、读取图像，并把图像转换为灰度图像并显示
    img = cv2.imread("D:\pycharm\pycharm_work\chepai\\num_for_car.jpg")  # 读取图片
    img_gray = cv2.cvtColor(img, cv2.COLOR_BGR2GRAY)  # 转换了灰度化
    cv2.imshow('gray', img_gray)  # 显示图片
    cv2.waitKey(0)
    # 2、将灰度图像二值化，设定阈值是100   转换后 白底黑字 ---》 目标黑底白字
    img_thre = img_gray
    # 灰点  白点 加错
    # cv2.threshold(img_gray, 130, 255, cv2.THRESH_BINARY_INV, img_thre)

    # 二值化处理 自适应阈值   效果不理想
    # th3 = cv2.adaptiveThreshold(img_gray, 255, cv2.ADAPTIVE_THRESH_GAUSSIAN_C, cv2.THRESH_BINARY, 11, 2)

    # 高斯除噪 二值化处理
    blur = cv2.GaussianBlur(img_gray, (5, 5), 0)
    ret3, th3 = cv2.threshold(blur, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)

    cv2.imshow('threshold', th3)
    cv2.imwrite('D:\pycharm\pycharm_work\chepai\wb_img.jpg', th3)
    cv2.waitKey(0)

The first line of imread(), because the flag is set to 1, reads a color image, and uses the cvtColor function to convert it into a grayscale image. If you read in a grayscale image, you can omit the second line of code, and then convert it into a binarization function. The threshold value of 180 can be modified, and then enhanced. The effect is as shown in the figure:

Three, single character cutting

The main strategy of single-character segmentation is to detect the total change of column pixels, because the area without characters is basically black, and the pixel value is low; the area with characters is more white, and the column pixel sum becomes larger!

The threshold of column pixel change is a problem. I see that many blogs use a fixed threshold for detection. Unless the binarized image after your processing is perfect, some images will be mixed into the white area and the segmentation will be wrong!

Considering that there are only 7 characters in the license plate, first determine the width. If it is less than one-seventh of the total width, it will be regarded as interference; in fact, it can be increased to one-eighth of the total width (because there may be connections in the middle of the license plate. symbol).

    n = 1
    start = 1
    end = 2
    temp = 1
    while n < width - 2:
        n += 1
        if (white[n] if arg else black[n]) > (0.05 * white_max if arg else 0.05 * black_max):
            # 上面这些判断用来辨别是白底黑字还是黑底白字
            # 0.05这个参数请多调整，对应上面的0.95
            start = n
            end = find_end(start, white, black, arg, white_max, black_max, width)
            n = end
            # 车牌框检测分割 二值化处理后 可以看到明显的左右边框  毕竟用的是网络开放资源 所以车牌框定位角度真的不准，
            # 所以我在这里截取单个字符时做处理，就当亡羊补牢吧
            # 思路就是从左开始检测匹配字符，若宽度（end - start）小与20则认为是左侧白条 pass掉  继续向右识别，否则说明是
            # 省份简称，剪切，压缩 保存，还有一个当后五位有数字 1 时，他的宽度也是很窄的，所以就直接认为是数字 1 不需要再
            # 做预测了（不然很窄的 1 截切  压缩后宽度是被拉伸的），
            # shutil.copy()函数是当检测到这个所谓的 1 时，从样本库中拷贝一张 1 的图片给当前temp下标下的字符
            if end - start > 5:  # 车牌左边白条移除
                print(" end - start" + str(end - start))
                if temp == 1 and end - start < 20:
                    pass
                elif temp > 3 and end - start < 20:
                    #  认为这个字符是数字1   copy 一个 32*40的 1 作为 temp.bmp
                    shutil.copy(
                        os.path.join("D:\pycharm\pycharm_work\chepai\fuzhi", "111.bmp"),  # 111.bmp 是一张 1 的样本图片
                        os.path.join("D:\pycharm\pycharm_work\chepai\\img_cut\\", str(temp) + '.bmp'))
                    pass
                else:
                    cj = th3[1:height, start:end]
                    cv2.imwrite("D:\pycharm\pycharm_work\chepai\img_cut_not_3240\\" + str(temp) + ".jpg", cj)

                    im = Image.open("D:\pycharm\pycharm_work\chepai\img_cut_not_3240\\" + str(temp) + ".jpg")
                    size = 32, 40
                    mmm = im.resize(size, Image.ANTIALIAS)
                    mmm.save("D:\pycharm\pycharm_work\chepai\img_cut\\" + str(temp) + ".bmp", quality=95)
                    temp = temp + 1

The cutting effect of the license plate is shown in the figure:

Four, single character recognition

            max1 = 0
            max2 = 0
            max3 = 0
            max1_index = 0
            max2_index = 0
            max3_index = 0
            for j in range(NUM_CLASSES):
                if result[0][j] > max1:
                    max1 = result[0][j]
                    max1_index = j
                    continue
                if (result[0][j] > max2) and (result[0][j] <= max1):
                    max2 = result[0][j]
                    max2_index = j
                    continue
                if (result[0][j] > max3) and (result[0][j] <= max2):
                    max3 = result[0][j]
                    max3_index = j
                    continue

            license_num = license_num + LETTERS_DIGITS[max1_index]
            print("概率：  [%s %0.2f%%]    [%s %0.2f%%]    [%s %0.2f%%]" % (
            LETTERS_DIGITS[max1_index], max1 * 100, LETTERS_DIGITS[max2_index], max2 * 100, LETTERS_DIGITS[max3_index],
            max3 * 100))

        print("车牌编号是: 【%s】" % license_num)

The final effect is shown in the figure:

Note: This picture is the running result of the following three programs. I have stitched the pictures together. .

train-license-province.py : Province abbreviation training recognition
train-license-letters.py : City code training recognition
train-license-digits.py : License plate number training recognition

finally

The license plate recognition cannot be 100% successful, but it can basically achieve a recognition degree of more than 95% through training. You can change the image in the capture_img file ( note that the image format and size will indirectly affect the recognition degree, and the license plate name is changed to car1.jpg ) is replaced with your own license plate to recognize the license plate through training.

Program framework click here to download

Just put the above test_images, train_images, train-saver, and the four core programs in according to the overall framework of the project ( the light green in the text is the download link )