End-to-end intelligent license plate recognition based on OpenCV+LPR model - deep learning and target detection algorithm application (including Python+Andriod all project source code)+CCPD data set

---

foreword

Based on the CCPD data set and LPR (License Plate Recognition, license plate recognition) model, combined with advanced technologies such as deep learning and target detection, this project builds a comprehensive license plate recognition system, realizing an end-to-end solution from license plate detection to character recognition plan.

First, we utilize the CCPD dataset, which contains a large number of Chinese license plate images, for model training and validation. The richness of this dataset helps the model better understand license plate features in different scenarios.

Next, we introduce the LPR model, which is a model specially designed for license plate recognition. Through deep learning technology, the LPR model can learn and recognize different types of license plates, whether it is a car, truck or motorcycle.

In the model design, we use object detection technology to allow the model to automatically locate and frame the license plate area in the image. In this way, the system can accurately find the license plate in different images, and it is not affected by different angles, lighting and other factors.

By extracting the license plate area, we input it into the LPR model for character recognition. The model recognizes the characters on the license plate, thus realizing a complete license plate recognition function.

Combining the above technologies, this project implements an end-to-end license plate recognition system that can detect and recognize license plates efficiently and accurately, no matter in different scenarios or in various environments. This system has important application value in traffic management, security monitoring and other fields.

overall design

This part includes the overall structure diagram of the system and the system flow chart.

System overall structure diagram

The overall structure of the system is shown in the figure.

System flow chart

The system flow is shown in the figure.

The algorithm process applied in the APP includes the algorithm description and process of rough license plate positioning, license plate fine positioning, fast tilt correction and non-segmented end-to-end character recognition, as shown in the figure below.

operating environment

This part includes Python environment, OpenCV environment and Andriod environment.

Python environment

Download the Miniconda 3.4.7.12 version from the Tsinghua TUNA open source mirror station https://mirrors.tuna.tsinghua.edu.cn/ , and select the corresponding version according to your own operating system.

Open Anaconda Prompt (miniconda3) in the start menu, enter the command line terminal, and enter the following command:

conda create -n tensorflow python=3.7#创建虛拟环境
conda install tensorflow-gpu #若没有独立GPU,可以选择安装CPU版本,去掉参数即可
conda install opencv-python=3.4.6
conda install pandas
#conda默认安装TensorFlow的2.1.0 stable版本
#其他常见的数据科学库如numpy在创建虚拟环境时或者安装TensorFlow
#若未安装,可以自行用conda或pip命令安装

OpenCV environment

Enter the release page https://opencv.org/releases/ of the OpenCV official website to download the Windows and Android packages of the OpenCV 3.4.6 version, and decompress them.

Android environment

Install the corresponding version of the development software as follows:

1. Development software and development kits

The download address is https://developer.android.google.cn/studio/ , APP development uses Android Studio 3.5.3 version.

Create NewProject, select EmptyActivity->next, enter the project name in the Name input box, name the java file package name in the project in the Package name input box, specify the project storage path in Save location, select [Language] ava, MinimumAPllevel specifies the minimum API version compatible with the project After setting, click the Finish button to complete the creation.

The Package name of this project is named com.pcr.lpr. After creating a project, you need to install SDK Platforms and SDKTools. Click the second icon from the upper right of the window to install.

SDK Platforms only select Android 9.0 (Pie), switch to the SDK Tools tab, and select the option circled in the figure below.

2. JDK settings

If the JDK is not installed in the system, you can go to https://www.oracle.com/java/technologies/javase-downloads.html to download it, click Android Studio, click the icon in the upper right box of the interface below to open the Project Structure, and fill in the JDK decompression folder path in JDK Location.

3. NDK settings

Download the Android-ndk-r14b version NDK from https://developer.android.google.cn/ndk/downloads/older_releases.html and decompress it .

Open Project Structure, and set the NDK path in the pop-up window, as shown in Figure 17-6.

module implementation

This project includes 3 modules: data preprocessing, model training, and APP construction. The functions and related codes of each module are introduced below.

1. Data preprocessing

Download the data set from the CCPD page https://github.com/detectRecog/CCPD , decompress and get the image data as shown in the figure.

After obtaining the dataset, preprocessing is performed. The license plate is cut according to the information provided by the image name of the data set, and the file name is divided according to "_". The fourth group of data is the coordinates of the four corners of the license plate in the picture, and the fifth group of data is the 7-digit license plate information. According to these two Group data can be divided into license plate pictures and license plate labels.

Both positive and negative samples have a size of 150X40 and are used for Cascade cascade classifier training. Negative samples can be cropped at will, and there is no license plate in the picture; positive samples are used to train unsegmented license plate character recognition. The relevant code is as follows:

#裁剪车牌
import cv2
import hashlib
import os, sys
import pandas as pd
import numpy as np
if not os.path.exists("output"):
    os.mkdir("output")
h = 40
w = 150
path = r"E:\ccpd_dataset\ccpd_base"  #数据集所在路径
filenames = os.listdir(path)
fp = open("pos_image.txt", "w", encoding="utf-8")
fn = open("neg_image.txt", "w", encoding="utf-8")
#用于存储图片的每一位车牌信息
df = pd.DataFrame(columns=('pic_name', '0', '1', '2', '3', '4', '5', '6'))
#正样本分割图片中的车牌
count = 0
#for img_name in filenames[0:40000]:
#为保证省份全覆盖，采用全部数据，而不是40000张
for img_name in filenames:
    #读取图片的完整名字
    image_path = os.path.join(path, img_name)
    image = cv2.imread(image_path
    #以“-”为分隔符，将图片名切分，其中iname[4]为车牌字符，iname[2]为车牌坐标
    iname = img_name.rsplit('/', 1)[-1].rsplit('.', 1)[0].split('-')
    plateChar = iname[4].split("_")
    #将文件名，七位车牌写入dataframe中
    new_line = [
        new_name, plateChar[0], plateChar[1], plateChar[2], plateChar[3],
        plateChar[4], plateChar[5], plateChar[6]
    ]
    df.loc[count, :] = new_line
    #crop车牌的左上角和右下角坐标
    [leftUp, rightDown] = [[int(eel) for eel in el.split('&')]
                           for el in iname[2].split('_')]
    #crop图片
    img = image[leftUp[1]:rightDown[1], leftUp[0]:rightDown[0]]
    height, width, depth = img.shape
    #将图片压缩成40*150，计算压缩比
    imgScale = h / height
    #(目标宽-实际宽)/2,分别向左、右拓宽，所有除以2
    deltaD = int((w / imgScale - width) / 2)
    #切割宽度向左平移，保证补够250
    leftUp[0] = leftUp[0] - deltaD
    #切割宽度向右平移，保证补够250
    rightDown[0] = rightDown[0] + deltaD
    #如果向左平移为负，坐标为0
    if (leftUp[0] < 0):
        rightDown[0] = rightDown[0] - leftUp[0]
        leftUp[0] = 0
    #按照高/宽 =40/150的比例切割,注意切割的结果不是40和250
    img = image[leftUp[1]:rightDown[1], leftUp[0]:rightDown[0]]
    newimg = cv2.resize(img, (w, h))  #resize成40*250
    new_name = 'pic' + str(count + 1).rjust(6, '0')
    cv2.imwrite("../output/pos/" + new_name + '.jpg', newimg)
    #将图片信息写入.txt文件中
    fp.write('pos/'+new_name +'.jpg'+'1 0 0 150 40' + plateChar + '\n')
    count += 1
df.to_csv('./pos.csv')
fp.close()
#负样本图片中没有车牌
count = 0
for img_name in filenames[400000:80000]:
    #补充完整图片路径
    image_path = os.path.join(path, img_name)
    image = cv2.imread(image_path)
    #裁剪不含车牌的区域
    new_img = image[0:40, 0:150]
    new_name = 'pic' + str(count + 40001).rjust(6, '0')
    cv2.imwrite("../output/neg/" + new_name + '.jpg', new_img)
    #将图片信息写入.txt文件中
    fn.write('neg/' + new_name + '.jpg' '\n')
    count += 1
fn.close()

The cropped positive samples are shown in the figure below.

The cropped negative samples are shown in the figure below.

2. Model training

The training of the cascade classifier and the convolutional neural network model without segmented license plate characters is as follows.

1) Train cascade classifiers

After getting the positive and negative samples, switch to the directory in the decompressed directory that downloaded OpenCV in the terminal \build\x64\vc15\bin\. Take D:\opencv\for example:

cd D:\opencv\build\x64\vc15\bin\

Execute the following command in the terminal, -infofill in the path of the positive sample.txt file; -bgfill in the path of the negative sample.txt file; -nummodify according to the number of positive samples obtained in the previous step; -W, -hare the width and height of the sample image respectively, all The width and height of the sample image must be the same, so fill in the width and height of the sample set in the previous step of cropping the license plate here.

opencv_createsamples.exe -vec pos.vec -info pos_image.txt -bg neg_image.txt -w 150 -h 40 -num 40000

After the execution is complete, generate the pos.vec file, and then execute the following command in the terminal:

opencv_traincascade.exe -data xml -vec pos.vec -bg neg_image.txt -numPos 40000 -numNeg 80000 -numStages 15 -precalcValBufSize 5000 -precalcIdxBufSize 5000 -w 150  -h 40  -maxWeakCount 200 -mode ALL -minHitRate 0.990 -maxFalseAlarmRate 0.20

Fill in -vecthe path of the pos.vec file generated in the previous step, -bgfill in the path of the negative sample.txt file, -numPosand -numNegmodify the number of positive samples obtained by splitting in the previous step. Other parameters can be provided by the command.

After the above command is executed, the trained cascade classifier file cascade.xml will be obtained in the sample directory.

2) Train the license plate character recognition model without segmentation

Non-segmented license plate character recognition is implemented using a convolutional neural network, and the relevant code is as follows:

import numpy as np
import pandas as pd
import pickle
import tensorflow as tf
from tensorflow.keras.layers import Dense, Conv2D, Flatten, MaxPooling2D
from tensorflow.keras.models import Model, Sequential
from tensorflow.keras.callbacks import TensorBoard
BATCH_SIZE = 64
def process_info(img_info):
    #处理每条csv中的信息，返回图片名称和one-hot后的标签
    img_path = img_info[0]
    label = np.zeros([238])  #7*34
    for i in range(7):
        index = int(img_info[i + 1])
        label[i * 34 + index] = float(1)
    return (img_path, label)
def parse_function(filename, label):
    #返回归一化后的解码图片矩阵和标签矩阵
    #读取文件
    image_string = tf.io.read_file(filename)
    #解码
    image_decoded = tf.image.decode_jpeg(image_string, channels=3)
    #变型
    image_resized = tf.image.resize(image_decoded, [150, 40])
    #归一化
    image_normalized = image_resized / 255.0
    return image_normalized, label
def create_dataset(filenames, labels):
    #创建数据集管道
    #补充完整的图片路径
    filenames = "D:/DL & ML/output/pos/" + filenames
    #从张量切片创建自定义数据集
    dataset = tf.data.Dataset.from_tensor_slices((filenames, labels))
    #并行处理
    dataset = dataset.map(parse_function,
                          num_parallel_calls=tf.data.experimental.AUTOTUNE)
    #数据集分批
    dataset = dataset.batch(BATCH_SIZE)
    #在训练时预先加载数据
    dataset = dataset.prefetch(buffer_size=tf.data.experimental.AUTOTUNE)
    return dataset
def macro_f1(y, y_hat, thresh=0.5):
    #计算一个批次的宏f1分数，返回宏f1分数
    y_pred = tf.cast(tf.greater(y_hat, thresh), tf.float32)
    tp = tf.cast(tf.math.count_nonzero(y_pred * y, axis=0), tf.float32)
    fp = tf.cast(tf.math.count_nonzero(y_pred * (1 - y), axis=0), tf.float32)
    fn = tf.cast(tf.math.count_nonzero((1 - y_pred) * y, axis=0), tf.float32)
    f1 = 2 * tp / (2 * tp + fn + fp + 1e-16)
    macro_f1 = tf.reduce_mean(f1)
    return macro_f1
def myModel():
	#自定义模型
model = Sequential()
#两层卷积一层池化
    model.add(Conv2D(64,(3,3),padding='same', activation='relu',input_shape=[150, 40, 3]))
    model.add(Conv2D(64, (3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D())
#两层卷积一层池化
    model.add(Conv2D(128, (3, 3), padding='same', activation='relu'))
    model.add(Conv2D(128, (3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D())
#两层卷积一层池化
    model.add(Conv2D(256, (3, 3), padding='same', activation='relu'))
    model.add(Conv2D(256, (3, 3), padding='same', activation='relu'))
    model.add(Conv2D(256, (3, 3), padding='same', activation='relu'))
    model.add(MaxPooling2D())
	#全连接层
    model.add(Flatten())
    model.add(Dense(1024, activation='relu'))
    model.add(Dense(512, activation='relu'))
    model.add(Dense(238, activation='sigmoid'))
    return model
if __name__ == "__main__":
    #读取数据以及预处理
    img_info = pd.read_csv(r"D:\DL & ML\output\pos.csv")
    train_pic_name = img_info.pic_name
    train_label = np.zeros([len(img_info), 238])
    for i in range(len(img_info)):
        _, train_label[i, ] = process_info(img_info.loc[i])
    #创建网络
    model = myModel()
    model.compile(loss='binary_crossentropy',
                  optimizer='adam',
                  metrics=[macro_f1])
    train_ds = create_dataset(train_pic_name, train_label)
    #调用tensorboard
    tb=TensorBoard(log_dir='logs')
    #训练网络
    model.fit(train_ds, epochs=30, callbacks=[tb])
    #保存模型
    model.save("segmention_free.h5")

After the training is completed, the model file is obtained in the directory where the code file is located segmention_free.h5.

3. APP construction

The relevant steps of APP construction are as follows.

1) Import the OpenCV library

Right-click on myApplicationk in the Project column on the left side of Android Studiot →New→Moduleto create a new module. Click ImportEclipseADTProjectto select sdk/javathe directory and click the next button to complete the import. Open the Project area on the left side of the interface openCVLibrary346\src\main\AndroidManifest.xml, and delete the text in the box in the figure below.

Open the newly introduced one in the Project area on the left Module\build.gradle, modify compileSdkVersionthe sum targetSdkVersionto 28, and modify the minSdkVersion to 19.

Open the Project area on the left app\build.gradle, and dependenciesadd the following statement in it:

implementation project(path:':openCVLibrary346')

2) Import the dynamic link library so file

Add the following statement in app\build.gradlethe android{defaultConfig{}}:

ndk {
    
    
	abiFilters 'armeabi -v7a'
}

Create app\src\main\jniLibsa directory, copy OpenCV-android-sdk-3.4.6\sdk\native\libsthe following into it.armeabi-v7jniLibs

3) Introduce C++ support and generate link library with CMake

Add the following statement in app\build.gradleandroid{defaultConfig{}}:

externalNativeBuild{
    
    
			cmake{
    
    
				cppFlags " - std=gnu++11"
			}
}

Create app\src\main\jniLibsa directory. UnderjniLibs Create javaWrapper.cpp, Import jni.hand string.h, all native functions nativedeclared with javawill be implemented in C++ in this file.

Create app\src\main\jniLibs\includethe directory under which all .h files are included. Create app\src\main\jniLibs\srcdirectory, javaWrapper.cppexcept , all .cpp files are included under this directory. Create a new File in the APP directory and name it CMakeLists.txt, and add the following code:

cmake_minimum_required(VERSION 3.4.1)
include_directories(src/main/jni/include)
include_directories(src/main/jni)
aux_source_directory(src/main/jni SOURCE_FILES)
aux_source_directory(src/main/jni/src SOURCE_FILES_CORE)
list(APPEND SOURCE_FILES ${
    
    SOURCE_FILES_CORE})
#修改为自己的opencv-android-sdk的jni路径
set(OpenCV_DIR D:\\Android\\OpenCV-android-sdk-3.4.6\\sdk\\native\\jni)
#查找包
find_package(OpenCV REQUIRED)
#添加库
add_library( #设置库的名称
        lpr
        SHARED
        ${
    
    SOURCE_FILES})
find_library( #设置变量路径名称
        log-lib
        log)
#将找到的库链接给lpr库
target_link_libraries( #指定目标库
        lpr
        ${
    
    OpenCV_LIBS}
        #将目标库链接到NDK中包含的日志库
            ${
    
    log-lib})
#在app\build.gradle的android{
    
     }中添加下列代码
externalNativeBuild {
    
    
        cmake {
    
    
            path "CMakeLists.txt"
        }
}

4. Import the trained model

Create a directory app\src\main\assets\lprand copy the trained model to the directory.

5. Register content provider and declare SD card access permission

Open it and add the following code app\src\AndroidManifest.xmlinside<application> </application>

<provider
            android:name="androidx.core.content.FileProvider"
            android:authorities="com.example.cameraalbumtest.fileprovider"
            android:exported="false"
            android:grantUriPermissions="true">
            <meta-data
                android:name="android.support.FILE_PROVIDER_PATHS"
                android:resource="@xml/file_paths" />
</provider>

app\src\resCreate the directory xml below file_paths.xmland modify the code in xml:

<?xml version="1.0" encoding="utf-8"?>
<paths xmlns:android="http://schemas.android.com/apk/res/android">
<external-path
    name="my_images"
    path=""/>
</paths>

The Name attribute can be set freely, and the path is empty, which means that the entire SD card will be shared, and it can also be modified to use the file address of the provider.

Open and add the following code app\src\AndroidManifest.xmloutside<application> </application>

<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />

6. Configure the Lite Pal database

Open app\build.gradleand add code dependenciesinside :

implementation "org.litepal.android:core:2.0.0"

app\src\main\assetsCreate inside Android Resource File(Android resource file), modify the code as:

<?xml version="1.0" encoding="utf-8"?>
<litepal>
    <dbname value="CarStore"></dbname>
    <version value="1"></version>
    <list>
        <mapping class="com.pcl.lpr.Cars"/>
    </list>
</litepal>

System test

This section contains training score and loss visualizations, APP test results.

1. Training Score and Loss Visualization

After the training of the unsegmented license plate character recognition model is completed, enter the following command in the terminal:

tensorboard --logdir logs # 其中--logdir填写logs目录的路径

In the curves of loss and macro_f1score, the horizontal axis is the number of training iterations, as shown in Figure 1 and Figure 2.

Figure 1 Training loss visualization

Figure 2 Training macro_f1 score visualization

Using HyperLPR to provide a trained model, the recognition accuracy rate is 95%~97%, and the download address is https://github.com/zeusees/HyperLPR/tree/master .

2. APP test results

The APP interface is shown in the figure below.

Click the camera icon in the main interface to enter the camera interface to take pictures with license plates, or click the picture icon in the main interface to enter and select pictures with license plates. Click the trash icon on the upper right to delete the imported pictures, and click the SUBMIT button to submit for detection, as shown in the figure below.

The image recognition results are shown in the figure below.

Click the icon at the top right of the "Results" page to save the recognition results. The saved records can be viewed by clicking the icon in the upper left corner of the "License Plate Recognition" page, as shown in the figure below.

A test example is shown in the figure.

Project source code download

See my blog resource download page for details

---

Other information download

If you want to continue to learn about artificial intelligence-related learning routes and knowledge systems, welcome to read my other blog " Heavy | Complete artificial intelligence AI learning-basic knowledge learning route, all materials can be downloaded directly from the network disk without paying attention to routines "
This blog refers to Github's well-known open source platform, AI technology platform and experts in related fields: Datawhale, ApacheCN, AI Youdao and Dr. Huang Haiguang, etc. There are about 100G related materials, and I hope to help all friends.