Opencv license plate recognition
Overview
The license plate recognition in this article is divided into several steps:
1. Image preprocessing
(1) Convert to grayscale image
(2) Perform Gaussian filtering
(3) Convert to binary image
(4) Edge detection
(5) Morphology deal with
2. Find the license plate
(1) Find the outline of each part in the preprocessed image
(2) Get the circumscribed rectangle of the outline
(3) Determine the rectangle as the license plate through the length and width conditions
3. Character segmentation
(1) License plate preprocessing
(2) Removing borders and rivets
(3) Vertical projection method to segment characters
4. Machine learning to recognize characters
The sample images processed in this article are as follows:
Each step is described in detail below:
1. Image preprocessing
The preprocessing function is as follows:
```cpp
Mat preprocessing(Mat input)
{
//转为灰度图
Mat gray;
cvtColor(input, gray, COLOR_BGR2GRAY, 1);
//进行高斯滤波
Mat gauss;
GaussianBlur(gray, gauss, Size(5, 5), 0, 0);
//阈值化操作
Mat thres_hold;
threshold(median, thres_hold, 100, 255, THRESH_BINARY);
//边缘检测
Mat canny_edge;
Canny(thres_hold, canny_edge, 50, 100, 5);
//形态学操作
int close_size = 5; //闭操作的核的大小
int open_size = 7; //开操作的核的大小
Mat element_close = getStructuringElement(MORPH_RECT, Size(1 + 2 * close_size, 1 + 2 * close_size));//定义闭操作的核
Mat element_open = getStructuringElement(MORPH_RECT, Size(1 + 2 * open_size, 1 + 2 * open_size));
Mat morph_close, morph_open;
morphologyEx(canny_edge, morph_close, MORPH_CLOSE, element_close);
morphologyEx(morph_close, morph_open, MORPH_OPEN, element_open);
return morph_open;
}
（1）转为灰度图
```cpp
Mat gray;
cvtColor(input, gray, COLOR_BGR2GRAY, 1);
Here, simply call the cvtcolor function to convert to grayscale image
(2) Perform Gaussian filtering
Mat gauss;
GaussianBlur(gray, gauss, Size(5, 5), 0, 0);
Simple Gaussian filtering. The kernel does not need to be too large to prevent the edges from being too blurry.
(3) Convert to binary image
Mat thres_hold;
threshold(median, thres_hold, 100, 255, THRESH_BINARY);
The purpose of binarization here is to roughly distinguish the foreground and background and highlight the license plate. Reduce interference items for subsequent edge detection.
(4) Edge detection
Mat canny_edge;
Canny(thres_hold, canny_edge, 50, 100, 5);
The ratio of high and low thresholds (third and fourth parameters) of the canny detection operator is generally 2:1, or 3:1. The detected
edge image is as follows:
(5) Morphological processing
int close_size = 5; //闭操作的核的大小
int open_size = 7; //开操作的核的大小
Mat element_close = getStructuringElement(MORPH_RECT, Size(1 + 2 * close_size, 1 + 2 * close_size));//定义闭操作核
Mat element_open = getStructuringElement(MORPH_RECT, Size(1 + 2 * open_size, 1 + 2 * open_size));//定义开操作核
Mat morph_close, morph_open;
morphologyEx(canny_edge, morph_close, MORPH_CLOSE, element_close);
morphologyEx(morph_close, morph_open, MORPH_OPEN, element_open);
Morphological closing and morphological opening operations are used here. The steps are to define the size of the operation core -> define the core -> perform the corresponding operation.
The purpose of the closing operation here is to fill the breaks in the outline and connect the broken edges into a whole. The larger the core, the larger the whole connected by the edges.
Result of the closing operation: 
The purpose of the opening operation is to eliminate the whole and line of some small pieces. The larger the core, the more obvious the effect.
Open operation result:
2. License plate extraction
Mat find_the_plate(Mat input)
{
Mat output; //提取出来的车牌
//寻找轮廓
vector<vector<Point>> contours; //定义轮廓点向量
vector<Vec4i> hierarchy; //轮廓的索引
findContours(input, contours, hierarchy, RETR_TREE, CHAIN_APPROX_SIMPLE);//提取轮廓
Point2f rect_info[4]; //用于储存最小矩形的四个顶点
RotatedRect minrect; //返回的最小外接矩形
int width; //最小矩形的宽
int height; //最小矩形的高
for (int i = 0; i < contours.size(); i++)
{
minrect = minAreaRect(contours[i]);
width = minrect.size.width;
height = minrect.size.height;
minrect.points(rect_info);
for (int j = 0; j < 4; j++)
{
if ( width / height >= 2 && width / height <= 5) //利用长宽比筛选条件
{
output = src(Rect(rect_info[1].x, rect_info[1].y, width, height));//根据筛选出的矩形提取车牌,相当于创建ROI
line(src, rect_info[j], rect_info[(j + 1) % 4], Scalar(0, 0, 255));//在原图画出符合条件的矩形
}
}
}
return output;
}
(1) Find the outline of each part in the preprocessed image
//寻找轮廓
vector<vector<Point>> contours; //定义轮廓点向量
vector<Vec4i> hierarchy; //轮廓的索引
findContours(input, contours, hierarchy, RETR_TREE, CHAIN_APPROX_SIMPLE);//提取轮廓
(Here input is the preprocessed image)
The second parameter contours is of type OutputArrayofArrays, which stores the found contours. Since the contour is originally a collection of points, that is, there is a vector, there are many contours.
Therefore, the third parameter hierarchy of vector<vertor> is of type OutputArrays and contains the topological information of the image. Each contour contours[i] contains 4 hierarchy elements, hierarchy[i][0]~
hierarchy[i][3], which respectively represent the index numbers of the next contour, the previous contour, the parent contour, and the embedded contour. So each element of hierarchy is of Vec4i type.
(2) Obtain the circumscribed rectangle of the outline and determine it to be the rectangle of the license plate through the length and width conditions
Point2f rect_info[4]; //用于储存最小矩形的四个顶点
RotatedRect minrect; //返回的最小外接矩形
int width; //最小矩形的宽
int height; //最小矩形的高
for (int i = 0; i < contours.size(); i++) //contours.size()轮廓的数量
{
minrect = minAreaRect(contours[i]);
width = minrect.size.width;
height = minrect.size.height;
minrect.points(rect_info);
for (int j = 0; j < 4; j++)
{
if ( width / height >= 2 && width / height <= 5) //利用长宽比筛选条件
{
output = src(Rect(rect_info[1].x, rect_info[1].y, width, height));//根据筛选出的矩形提取车牌,相当于创建ROI
line(src, rect_info[j], rect_info[(j + 1) % 4], Scalar(0, 0, 255));//在原图画出符合条件的矩形
}
}
}
return output;
}
Define minrect of type RotatedRect to store the rectangle, and the minAreaRect function stores the circumscribed rectangle of the outline into minrect.
The For loop visits each enclosing rectangle. If the aspect ratio of the enclosing rectangle meets the filtering conditions (for example, 2<y:x<5), it will be identified as a license plate and marked with a red frame in the original image.
The result is shown below:
3. Character segmentation
Here we need to first explain the principle of vertical projection character segmentation:
after binarizing the character graphics, the image will become an image with a black background and white characters. At this time, traverse the entire image and count the number of white pixels in each column. Then draw a histogram of the number of white pixels in each column.
(As shown in the figure below, the horizontal axis is each column, and the vertical axis is the number of white pixels on the column)
At this time, the column without white pixels is the dividing line between characters, and you only need to know which column it is.
void character_division(Mat input)
{
Mat expansion; //车牌图片放大
resize(input, expansion, Size(input.cols * 3, input.rows * 3), 0, 0);
int width = expansion.size().width; //获取图像的长和宽
int height = expansion.size().height;
Mat gray;
cvtColor(expansion, gray, CV_BGR2GRAY, 1);
//中值滤波
medianBlur(gray, gray, 3); //核太大会使得二值化后的字变模糊
Mat thres_hold;
threshold(gray, thres_hold, 0,255, THRESH_OTSU);
/*
int dilate_size = 2; //开操作的核的大小
Mat element_open = getStructuringElement(MORPH_RECT, Size(1 + 2 * dilate_size, 1 + 2 * dilate_size));
Mat morph_dilate;
morphologyEx(thres_hold, morph_dilate, MORPH_DILATE, element_open);
*/
int pixelvalue; //每个像素的值
int * white_nums = new int[width](); //定义动态数组并初始化， 储存每一列的白色像素数量
//遍历每一个像素，去掉边框和铆钉，再统计剩下每一列白色像素的数量
for (int col = 0; col < width; col++)
{
/*去除竖直的边框和铆钉*/
int cols_convert_num = 0;//每一列黑白转变的次数
for (int i = 0; i < height - 1; i++)//遍历某一列的所有元素，计算转变次数
{
if (thres_hold.at<uchar>(i, col) != thres_hold.at<uchar>(i + 1, col))
cols_convert_num++;
}
if (cols_convert_num < cols_thres_value)
{
continue;
}
/*去除竖直的边框和铆钉*/
for (int row = 0; row < height; row++)
{
/*去除水平的边框和铆钉*/
int rows_convert_num = 0;//每一行黑白转变的次数
for (int j = 0; j < width - 1; j++)//遍历某一行的所有元素，计算转变次数
{
if (thres_hold.at<uchar>(row, j) != thres_hold.at<uchar>(row, j + 1))
rows_convert_num++;
}
if (rows_convert_num < rows_thres_value)
{
continue;
}
/*去除水平的边框和铆钉*/
pixelvalue = thres_hold.at<uchar>(row, col);
if (pixelvalue == 255)
white_nums[col]++; //统计白色像素的数量
}
}
//画出投影图
Mat verticalProjection(height,width, CV_8UC1, Scalar(0,0,0));
for (int i = 0; i < width; i++)
{
line(verticalProjection,Point(i, height),Point(i, height - white_nums[i]), Scalar(255,255,255));
}
//根据投影图进行分割
vector<Mat> character;
int character_num = 0; //字符数量
int start; //进入字符区的列数
int end; //退出字符区的列数
bool character_block = false;// 是否进入了字符区
for (int i = 0; i < width; i++)
{
if(white_nums[i] != 0 && character_block == false) //刚进入字符区
{
start = i;
character_block = true;
}
else if (white_nums[i] == 0 && character_block == true) //刚出来字符区
{
character_block = false;
end = i;
if (end - start >= 6)
{
Mat image = expansion(Range(0, height), Range(start, end));
character.push_back(image); //push.back适用于vector类型 在数组尾部添加一个数据
}
}
}
delete[] white_nums; //删除动态数组
imshow("pro", thres_hold);
imshow("projection", verticalProjection);
}
(1) Preprocessing of license plate images
Mat expansion; //车牌图片放大
resize(input, expansion, Size(input.cols * 3, input.rows * 3), 0, 0);
int width = expansion.size().width; //获取图像的长和宽
int height = expansion.size().height;
Mat gray;
cvtColor(expansion, gray, CV_BGR2GRAY, 1);
//中值滤波
medianBlur(gray, gray, 3); //核太大会使得二值化后的字变模糊
Mat thres_hold;
threshold(gray, thres_hold, 0,255, THRESH_OTSU);
This process includes:
1. Use the resize function to enlarge the license plate, and note the enlarged width and height to facilitate subsequent observation and processing.
2. Convert to grayscale image
3. Median filtering. Prevent the interference of salt and pepper noise.
4. Binarization. The OTSU algorithm (maximum inter-class difference algorithm) is used here. The advantage of the OTSU algorithm is that you do not need to set the threshold yourself. The algorithm will automatically calculate the threshold based on the image, so the third parameter is meaningless. But sometimes the foreground and background are reversed.
Preprocessed image:
It can be seen that the license plate at this time not only has characters, but also borders and rivets, which will have an impact on the subsequent vertical projection segmentation of characters. So the next step is to remove these interference items .
(2) Remove borders and rivets.
Let me talk about it first, which is an array used to store the white pixels of each column. Because the values of array elements are dynamic, you need to use new to create the array first, and then use delete to release the array when you are finished.
#define rows_thres_value 10 //除掉车牌框时，行跳变的阈值
#define cols_thres_value 2 //除掉车牌框时，列跳变的阈值
int pixelvalue; //每个像素的值
int * white_nums = new int[width](); //定义动态数组并初始化， 储存每一列的白色像素数量
//遍历每一个像素，去掉边框和铆钉，再统计剩下每一列白色像素的数量
for (int col = 0; col < width; col++)
{
/*去除水平的边框和铆钉*/
int cols_convert_num = 0;//每一列黑白转变的次数
for (int i = 0; i < height - 1; i++)//遍历某一列的所有元素，计算转变次数
{
if (thres_hold.at<uchar>(i, col) != thres_hold.at<uchar>(i + 1, col))
cols_convert_num++;
}
if (cols_convert_num < cols_thres_value)
{
continue;
}
/*去除水平的边框和铆钉*/
for (int row = 0; row < height; row++)
{
/*去除竖直的边框和铆钉*/
int rows_convert_num = 0;//每一行黑白转变的次数
for (int j = 0; j < width - 1; j++)//遍历某一行的所有元素，计算转变次数
{
if (thres_hold.at<uchar>(row, j) != thres_hold.at<uchar>(row, j + 1))
rows_convert_num++;
}
if (rows_convert_num < rows_thres_value)
{
continue;
}
/*去除竖直的边框和铆钉*/
pixelvalue = thres_hold.at<uchar>(row, col);
if (pixelvalue == 255)
white_nums[col]++; //统计白色像素的数量
}
}
The principle of removing borders and rivets here is: Take row as an example. Count the number of white and black transitions that occur in each line, because in the character area, the number of transitions must be more, while the number of transitions in the background and border areas is smaller. Simply set a threshold and rows with fewer transitions than the threshold will not be counted in calculating white pixels. The same goes for columns.
/*去除竖直的边框和铆钉*/
int rows_convert_num = 0;//每一行黑白转变的次数
for (int j = 0; j < width - 1; j++)//遍历某一行的所有元素，计算转变次数
{
if (thres_hold.at<uchar>(row, j) != thres_hold.at<uchar>(row, j + 1))//本行与下一行颜色不同
rows_convert_num++;
}
if (rows_convert_num < rows_thres_value)
{
continue;
}
/*去除竖直的边框和铆钉*/
Take behavior as an example. From the first row j = 0, to the penultimate row j = width - 1.
If the color of this row is different from that of the next row, the number of changes will be +1
if (thres_hold.at<uchar>(row, j) != thres_hold.at<uchar>(row, j + 1))//本行与下一行颜色不同
rows_convert_num++;
If the number of transitions is less than the threshold, this column will not be included in statistics.
if (rows_convert_num < rows_thres_value)
{
continue;
}
(3) Draw the projection diagram and segment it.
Mat verticalProjection(height,width, CV_8UC1, Scalar(0,0,0));
for (int i = 0; i < width; i++)
{
line(verticalProjection,Point(i, height),Point(i, height - white_nums[i]), Scalar(255,255,255));
}
//根据投影图进行分割
vector<Mat> character;
int character_num = 0; //字符数量
int start; //进入字符区的列数
int end; //退出字符区的列数
bool character_block = false;// 是否进入了字符区
for (int i = 0; i < width; i++)
{
if(white_nums[i] != 0 && character_block == false) //刚进入字符区
{
start = i;
character_block = true;
}
else if (white_nums[i] == 0 && character_block == true) //刚出来字符区
{
character_block = false;
end = i;
if (end - start >= 6)
{
Mat image = expansion(Range(0, height), Range(start, end));
character.push_back(image); //push.back适用于vector类型 在数组尾部添加一个数据
}
}
}
delete[] white_nums; //删除动态数组
The drawn projection diagram is as follows:
The algorithm here is: use the method to determine whether the height is 0, use the bool variable to mark whether it is the character area, and use start and end to record the number of columns entering the character area.
Then store the image of a certain character range into the character array.
if (end - start >= 6)
{
Mat image = expansion(Range(0, height), Range(start, end));
character.push_back(image); //push.back适用于vector类型 在数组尾部添加一个数据
}
4. Machine learning to recognize characters
Studying, will deal with it later