OpenCV C++ case practice thirty-three "Defect Detection"
Preface
This case will use OpenCV C++ to detect PCB printing defects. At present, defect detection algorithms can be divided into two major categories:
one: defect detection based on template matching;
two: defect detection based on deep learning, which mainly uses target detection to identify defective parts.
The algorithm in this article is mainly based on template matching algorithm for defect detection. Please refer to the article "Printing Label Defect Detection Algorithm Based on Difference Model" to reproduce the algorithm. Interested friends can read the original article.
1. Result Demonstration
2. Defect detection algorithm
2.1. Multiple template images
Qualified label images are collected through industrial cameras as a training data set for the difference model. One of the qualified label images is selected to perform Gaussian smoothing, grayscale erosion and grayscale expansion operations to obtain a multivariate template image for training the difference model.
Convolve the qualified image f(x, y) with the Gaussian kernel filter to obtain the Gaussian smooth image f1(x, y). Construct a rectangular structural element with a size of 11×11, perform a grayscale corrosion operation on the qualified label image, and obtain a grayscale corrosion image f2 (x, y). Then construct a 13×13 rectangular structural element, perform grayscale expansion operation on the qualified label image [3], and obtain the grayscale expansion image f3 (x, y).
2.2. Training difference model
The multivariate template images f1 (x, y), f 2 (x, y) and f 3 (x, y) are used as training data sets to train the difference model. Calculate the average and standard deviation of the pixels at the same coordinates of all images [4] to obtain the average image F (x, y):
Standard deviation image V(x,y):
In this article, F (x, y) and V (x, y) are the standard images and difference images in the training process of the difference model.
In order to adapt the ideal difference model to the normal process error range, the relative threshold VarThreshold=[bu, bl] parameter is added. Among them, bu is the upper limit relative threshold, and bl is the lower limit relative threshold. as shown in picture 2. Then the two threshold images T u, l (x, y) are calculated as follows:
bright threshold image: Tu (x, y) = F (x, y) + bu* V (x, y)
dark threshold image: Tl (x ,y)=F(x,y)-bl* V(x,y)
Compare the gray value between pixels between the registered image c(x, y) and the threshold image Tu, l(x, y) of the difference model. When the following conditions are met, it is detected. to the defective area.
c(x,y)>Tu(x,y)∨c(x,y)<T l(x,y)
3. Image Registration
As shown in the picture is the template image
As shown in the figure is the image to be detected, we need to perform image registration between the image to be detected and the template image. Here I use image affine transformation to correct the two images. I won’t go into details about image correction. You can refer to my blog post OpenCV C++ Case Practice 4 "Image Perspective Correction" . Go directly to the code here
3.1 Function source code
//图像定位矫正
bool ImageLocal(cv::Mat srcImg, cv::Mat& warpImg, Point2f SrcAffinePts[])
{
Mat grayImg;
if (srcImg.channels() != 1)
{
cvtColor(srcImg, grayImg, COLOR_BGR2GRAY);
}
else
{
grayImg = srcImg.clone();
}
Mat blurImg;
medianBlur(grayImg, blurImg, 5);
Mat binImg;
threshold(blurImg, binImg, 10, 255, THRESH_BINARY);
//namedWindow("binImg", WINDOW_NORMAL);
//imshow("binImg", binImg);
vector<vector<Point>>contours;
findContours(binImg, contours, RETR_EXTERNAL, CHAIN_APPROX_SIMPLE);
RotatedRect bRect;
for (int cnt = 0; cnt < contours.size(); cnt++)
{
double area = contourArea(contours[cnt]);
if (area > 1000)
{
bRect = minAreaRect(contours[cnt]);
}
}
if (bRect.size.empty())return false;//如果没有找到最小外接矩形，返回false
//找到最小外接矩形四个顶点
Point2f srcPoints[4];
bRect.points(srcPoints);
//for (int i = 0; i < 4; i++)
//{
// line(srcImg, srcPoints[i], srcPoints[(i + 1) % 4], Scalar(0, 255, 0), 3);
//}
//将四个点按照左上、右上、右下、左下进行区分
int TL, TR, BR, BL;
double addmax = 0.0, addmin = 999.9, submax = 0.0, submin = 999.9;
for (int i = 0; i < 4; i++)
{
double addval = srcPoints[i].x + srcPoints[i].y;
double subval = srcPoints[i].x - srcPoints[i].y;
if (addval > addmax)
{
addmax = addval;
BR = i;
}
if (addval < addmin)
{
addmin = addval;
TL = i;
}
if (subval > submax)
{
submax = subval;
TR = i;
}
if (subval < submin)
{
submin = subval;
BL = i;
}
}
double LeftHeight = EuDis(srcPoints[TL], srcPoints[BL]);
double RightHeight = EuDis(srcPoints[TR], srcPoints[BR]);
double MaxHeight = max(LeftHeight, RightHeight);
double UpWidth = EuDis(srcPoints[TL], srcPoints[TR]);
double DownWidth = EuDis(srcPoints[BL], srcPoints[BR]);
double MaxWidth = max(UpWidth, DownWidth);
//这里使用的顺序是左上、右上、右下、左下顺时针顺序。SrcAffinePts、DstAffinePts要一一对应
SrcAffinePts[0] = Point2f(srcPoints[TL]);
SrcAffinePts[1] = Point2f(srcPoints[TR]);
SrcAffinePts[2] = Point2f(srcPoints[BR]);
SrcAffinePts[3] = Point2f(srcPoints[BL]);
Point2f DstAffinePts[4] = {
Point2f(0,0),Point2f(MaxWidth,0),Point2f(MaxWidth,MaxHeight),Point2f(0,MaxHeight) };
Mat M = getPerspectiveTransform(SrcAffinePts, DstAffinePts);
warpPerspective(srcImg, warpImg, M, Size(MaxWidth, MaxHeight), 1, 0, Scalar::all(0));
return true;
}
3.1 Functional effects
4. Multiple template images
How to calculate the mean image, difference image, and bright and dark threshold images is reproduced in the source code below. Please read the source code for details.
4.1 Function source code
//计算均值图像
void meanImage(cv::Mat gaussianImg, cv::Mat erodeImg, cv::Mat dilateImg, cv::Mat& meanImg)
{
meanImg = Mat::zeros(gaussianImg.size(), CV_8U);
for (int i = 0; i < gaussianImg.rows; i++)
{
uchar* gData = gaussianImg.ptr<uchar>(i);
uchar* eData = erodeImg.ptr<uchar>(i);
uchar* dData = dilateImg.ptr<uchar>(i);
uchar* mData = meanImg.ptr<uchar>(i);
for (int j = 0; j < gaussianImg.cols; j++)
{
mData[j] = (gData[j] + eData[j] + dData[j]) / 3;
}
}
}
//计算差异图像
void diffImage(cv::Mat gaussianImg, cv::Mat erodeImg, cv::Mat dilateImg, cv::Mat meanImg, cv::Mat& diffImg)
{
diffImg = Mat::zeros(gaussianImg.size(), CV_8U);
for (int i = 0; i < gaussianImg.rows; i++)
{
uchar* gData = gaussianImg.ptr<uchar>(i);
uchar* eData = erodeImg.ptr<uchar>(i);
uchar* dData = dilateImg.ptr<uchar>(i);
uchar* mData = meanImg.ptr<uchar>(i);
uchar* Data = diffImg.ptr<uchar>(i);
for (int j = 0; j < gaussianImg.cols; j++)
{
Data[j] = sqrt(powf((gData[j] - mData[j]), 2) + powf((eData[j] - mData[j]), 2) + powf((dData[j] - mData[j]), 2) / 3.0);
}
}
}
//计算亮、暗阈值图像
void threshImg(cv::Mat meanImg, cv::Mat diffImg,cv::Mat &LightImg,cv::Mat& DarkImg)
{
double bu = 1.2;
double bl = 0.8;
Mat mul_bu, mul_bl;
multiply(diffImg, bu, mul_bu);
multiply(diffImg, bl, mul_bl);
LightImg = Mat::zeros(meanImg.size(), CV_8U);
DarkImg = Mat::zeros(meanImg.size(), CV_8U);
for (int i = 0; i < meanImg.rows; i++)
{
uchar* mData = meanImg.ptr<uchar>(i);
uchar* dData = diffImg.ptr<uchar>(i);
uchar* lData = LightImg.ptr<uchar>(i);
uchar* DData = DarkImg.ptr<uchar>(i);
uchar* buData = mul_bu.ptr<uchar>(i);
uchar* blData = mul_bl.ptr<uchar>(i);
for (int j = 0; j < meanImg.cols; j++)
{
lData[j] = saturate_cast<uchar>(mData[j] + buData[j]);
DData[j] = saturate_cast<uchar>(mData[j] - blData[j]);
}
}
}
The picture below is a bright threshold image.
The picture below is a dark threshold image.
5. Defect detection
Above, we calculated the bright and dark threshold images of the template, mainly by comparing them with the gray values of the two images to determine the defective part.
As shown in the figure: Compare the gray value between pixels between the registered image c(x, y) and the threshold image Tu, l(x, y) of the difference model. When the following conditions are met , which is the detected defective area.
c(x,y)>Tu(x,y)∨c(x,y)<T l(x,y)
Since the defects extracted at this time are based on affine correction, if the results need to be displayed on the original image, the detection results need to be inversely transformed. Please read the source code for details.
5.1 Function source code
//缺陷检测
void DetectImg(cv::Mat warpImg,cv::Mat LightImg, cv::Mat DarkImg, Point2f SrcAffinePts[],cv::Mat decImg, cv::Mat& showImg)
{
int th = 10;//容差阈值
Mat resImg = Mat::zeros(warpImg.size(), CV_8U);
for (int i = 0; i < warpImg.rows; i++)
{
uchar* sData = warpImg.ptr<uchar>(i);
uchar* lData = LightImg.ptr<uchar>(i);
uchar* dData = DarkImg.ptr<uchar>(i);
uchar* rData = resImg.ptr<uchar>(i);
for (int j = 0; j < warpImg.cols; j++)
{
//识别缺陷
if ((sData[j]-th) > lData[j]||(sData[j]+th) < dData[j])
{
rData[j] = 255;
}
}
}
Mat kernel = getStructuringElement(MORPH_RECT, Size(3, 3));
morphologyEx(resImg, resImg, MORPH_OPEN, kernel);
kernel = getStructuringElement(MORPH_RECT, Size(7, 7));
dilate(resImg, resImg, kernel);
//namedWindow("resImg", WINDOW_NORMAL);
//imshow("resImg", resImg);
//绘制缺陷结果
vector<vector<Point>>contours;
findContours(resImg, contours, RETR_LIST, CHAIN_APPROX_SIMPLE);
for (int t = 0; t < contours.size(); t++)
{
if (contourArea(contours[t]) > 50)
{
Rect rect = boundingRect(contours[t]);
rectangle(showImg, rect, Scalar(0, 0, 255), 2);
}
}
//将结果反变换回原图像
Point2f DstAffinePts[4] = {
Point2f(0,0),Point2f(decImg.cols,0),Point2f(decImg.cols,decImg.rows),Point2f(0,decImg.rows) };
Mat M = getPerspectiveTransform( DstAffinePts, SrcAffinePts);
warpPerspective(showImg, showImg, M, decImg.size(), 1, 0, Scalar::all(0));
}
6. Effect Demonstration
As shown in the effect above, compared with the template image, basically all defects in the image to be tested are detected, and there are very few false detections. When applied to different object detection, you need to make slightly smaller parameter adjustments based on your own image data. Here I just provide you with an algorithm idea, and everyone is welcome to communicate and learn! ! !
Summarize
This article uses OpenCV C++ to detect PCB printing defects. The main operations are as follows.
1. Perform affine transformation on the image and register it with the template image.
2. Calculate the difference image to obtain the light and dark threshold images based on the template.
3. Compare the image to be detected with the light and dark threshold images pixel by pixel, and set the threshold. Anything beyond the threshold is a defect