Computer Vision (OpenCV+TensorFlow

Computer Vision (OpenCV+TensorFlow)

Article directory

Computer Vision (OpenCV+TensorFlow)
Preface
3. Image processing
Summarize

Preface

本系列文章是上篇OpenCV库操作的后续，主要内容还是有关图像的处理，建议先看完上篇内容后再来食用，如果你要问为什么不搞一个合集，那我只能说苦逼大学生没有一天空闲时间，只能某天抽出一点时间去更新（虽然我的更新频率也不是很快）

3. Image processing

1. Image smoothing

If the incoming image has some particularly bright points or other points that can interfere with image detection, we must first perform image smoothing before image detection to minimize the interference effect of these points.

1.1.Mean filter

Mean filtering: We must first specify a convolution kernel. This convolution kernel is actually a square matrix and then divided by the number of elements in the matrix. Let's take a 3x3 matrix as an example. Replace the target pixel value with the average of all pixel values in the convolution kernel window. For the convolution kernel matrix in the above figure, 58 is the center point, and it should be replaced with the average of these nine numbers.

# img就是图像，后面是卷积核的维度,blur就是处理过的图像
blur=cv2.blur(img,(3,3))

Advantages and disadvantages of mean filtering:

Mean filtering has the advantages of simple operation, high efficiency and easy implementation, and can obtain a rough description of object characteristics.
Mean filtering cannot protect image details well. While denoising the image, it also destroys image details and loses image feature information (this is not difficult to understand, because mean filtering calculates based on the mean and only replaces the center value. )

1.2 Box filtering

Basically the same as mean filtering, you can choose whether to perform normalization. Normalization means summing and dividing numbers, and not normalizing means just summing
If If the sum exceeds 255, just take 255

blur=cv2.blur(img,(3,3),normalize=False)

1.3 Gaussian filtering

** Have you heard of the Gaussian function? The Gaussian function is the normal distribution function. We can think of it this way. In the mean filtering above, we add up all the values in the convolution kernel and divide it by 9. But for the center point, not all points have the same impact on it. There must be some points. It has a great impact on him, and some points have almost no impact. Now, do you feel that the mean filter treats the impact of all points on the center point as the same, which seems unreasonable now. We should add a weight to each number, so that the numbers with greater influence have greater weight and have greater impact on the center. The Gaussian function shows that the closer to the center point, the greater the impact on the center point**

# 前两个参数就不说了，后面的那个1表示高斯核函数在X方向的的标准偏差，我们这里没有写y方向的，那么
# x方向和y方向一致，如果σ较小，这样对图像的平滑效果就不是很明显；
# 相反，σ较大时，比较类似于均值模板，对图像的平滑效果就比较明显。至于具体作用，可以看一下高斯函数
cv2.GaussianBlur(img,(5,5),1)

1.4 Median filtering

In fact, the operation is the same as above. We first take a convolution kernel, but the replacement number is the median of all elements in the convolution kernel.

# 5就是卷积核的长度和宽度，必须是比1大的奇数
cv2.medianBlur(img,5)

2. Image form conversion

This part of the content mainly involves the knowledge of morphology, and the basic idea of morphology is to use structural elements of a certain form to measure and extract the corresponding shapes in the image to achieve the purpose of analysis and identification.

2.1 Corrosion operation

The function of corrosion: eliminate the boundary points of objects, shrink the boundaries inward, and remove objects smaller than structural elements. Can separate two objects with small connections. This method can be used to remove burrs, small protrusions, etc. If there is a small connection between two objects, when the structure is large enough, the two objects can be separated

# 这里我们先导进来一张图像
img = cv2.imread('dige.png')

cv2.imshow('img', img)
cv2.waitKey(0)
cv2.destroyAllWindows()
# 我们可以看到这里面有许多毛刺，我们现在就要给这个图片加上黑色的背景，把这些毛刺“腐蚀掉”

# 我们在这里先创建了一个3x3的矩阵，矩阵的元素都为1，这个矩阵就是我们的结构元素，
# 通过这个矩阵去腐蚀
kernel = np.ones((3,3),np.uint8) 
# 这个erode函数就是腐蚀函数，img就是图片，kernel就是用于腐蚀的结构元素，iterations就是腐蚀的次数
erosion = cv2.erode(img,kernel,iterations = 1)
# 我们看结果，毛刺被去除了。
cv2.imshow('erosion', erosion)
cv2.waitKey(0)
cv2.destroyAllWindows()

The corrosion operation is that the structural element matrix moves sequentially on the image matrix, and the minimum value of the element covered by the structural element matrix at each position replaces the center position value of B. Then we know that in the grayscale image, the darker the point, the smaller the value, and the brighter the point, the larger the value. Then we can briefly summarize it again
The corrosion operation is targeted at the white elements in the image , if we want to remove the white elements in the image that interfere with the image, we can use the erosion operation

2.2 Expansion operation

We can compare the expansion operation to the corrosion operation, which is to make the white elements larger, just like expansion. The corrosion operation is to take the minimum value, and the expansion operation is to take the maximum value.

# 这个我们还是把刚才的照片导进来，然后进行一个腐蚀操作
img = cv2.imread('dige.png')
cv2.imshow('img', img)
cv2.waitKey(0)
cv2.destroyAllWindows()
kernel = np.ones((3,3),np.uint8) 
dige_erosion = cv2.erode(img,kernel,iterations = 1)

cv2.imshow('erosion', erosion)
cv2.waitKey(0)
cv2.destroyAllWindows()
# 然后我们对进行腐蚀操作后的图像进行膨胀操作，dilate()就是膨胀，与腐蚀函数基本一致
kernel = np.ones((3,3),np.uint8) 
dige_dilate = cv2.dilate(dige_erosion,kernel,iterations = 1)

cv2.imshow('dilate', dige_dilate)
cv2.waitKey(0)
cv2.destroyAllWindows()

2.3 Opening and closing operations

The opening operation is to corrode first and then expand: after we corrode, although the burrs are removed, the corresponding content part also becomes smaller, because we use the matrix to corrode, and the burrs are all irregular shapes. Therefore, the main content part will definitely be reduced by a part, and the expansion operation can make up for the reduced part.
**The closing operation is to expand first and then erode: **It is useful when filling small holes in foreground objects or small black spots on objects

# 开：先腐蚀，再膨胀
img = cv2.imread('dige.png')

kernel = np.ones((5,5),np.uint8) 
# morphology就是形态学的意思
opening = cv2.morphologyEx(img, cv2.MORPH_OPEN, kernel)

cv2.imshow('opening', opening)
cv2.waitKey(0)
cv2.destroyAllWindows()

# 闭：先膨胀，再腐蚀
img = cv2.imread('dige.png')

kernel = np.ones((5,5),np.uint8) 
closing = cv2.morphologyEx(img, cv2.MORPH_CLOSE, kernel)

cv2.imshow('closing', closing)
cv2.waitKey(0)
cv2.destroyAllWindows()

2.4 Gradient operation

Before calculating this gradient, we need to first supplement the concept of image gradient.

Explanation of image gradient:https://blog.csdn.net/literacy_wang/article/details/109640726

image gradient

Let’s first review the concept of gradient in advanced mathematics

The gradient is not a real number, it is a vector with magnitude and direction. Now take a binary function as an example. Assume a binary function f(x,y). The gradient at a certain point is:

is actually the directional derivative. The direction of the gradient is the direction in which the function changes the fastest. It is easy to find the maximum value along the direction of the gradient.
Sometimes, the outline of an object in a blurry image is not very obvious, and the grayscale change at the edge of the outline is not very strong, which leads to a weak sense of layering when we look at it, but when it is clear, The grayscale changes at the edges of the object contours in the image are obvious and have a strong sense of layering. So how to define whether this grayscale change is obvious or not?

You can use derivatives. The definition of derivatives is the rate of change of a function at a certain point, and the direction of the gradient is the direction in which the function changes fastest. So if we regard the image as a two-dimensional discrete function, then the image gradient is the derivative of this two-dimensional discrete function.

As can be seen from the picture above, if the adjacent gray values of an image change, then the gradient exists. If the adjacent pixels of the image do not change, then the gradient is 0
Add the gradient value and the corresponding pixel, then if the gray value does not change, the pixel will not change. When the gray value is traversed, the pixel value will also change.
Let’s look at it. After adding the original value and the corresponding position of the gradient, the difference between the original image pixel points 100 and 90 is only 10, but now it is 110 and 90, the difference is 20. The contrast is obviously enhanced. In particular, the outlines and edges of objects in the image greatly enhance the difference from the background. This is the principle of gradient enhancement of the image.

Through the above explanation, I believe everyone has understood the image gradient, so here is a summary
The image gradient operation is simply to amplify the difference between pixels. In this way, the layering of the image is enhanced

OpenCV calculates image gradient

# 导进来一张图片，是一个圆
pie = cv2.imread('pie.png')
# 还是这个形态学操作，只是其中一个参数换成了cv2.MORPH_GRADIENT
kernel = np.ones((7,7),np.uint8) 
gradient = cv2.morphologyEx(pie, cv2.MORPH_GRADIENT, kernel)

cv2.imshow('gradient', gradient)
cv2.waitKey(0)
cv2.destroyAllWindows()

We see that the final structure is a ring. In fact, it should be called the outline of a circle more appropriately. The reason for this is also very simple. We can just print the image. As shown in the picture below, you can find that many are 0, but there are also many 0, it’s just that there is too much data and it is not fully displayed. The array below is the gradient array of the source image. Because the source image, except for the circles, is all black, then the gradient of these blacks must be 0. There is also the center part of the circle, which is all white and the subtraction is also 0. Then there is a gradient. There is only the outline of a circle. Through this case, we can also see that the image formed by the gradient array of an image is the outline of the image.

Top hat and black hat

Top hat = source image - opening operation. After opening operation, we remove the burrs. Then subtract the image after opening operation from the source image, then what is left is just burrs
Black hat = closed operation - source image. After the closed operation, the burr is still there, but the image after the closed operation is a little fatter than the source image. After subtracting the source image, the thorn is gone, leaving only the closed operation larger than the source image. That part, it’s like getting a small outline without burrs, but the outline is not very clear
Now we can’t see the role of top hat and black hat. We can learn more later. See, here you can read an article to feel the effect

Politeness and the role of black hat:https://blog.csdn.net/great_yzl/article/details/119594716

For the gradient calculation of various complex images, it is impossible to accurately calculate the gradient based on morphology alone. This is also one of the backgrounds for the emergence of many gradient operators. Next, we will introduce various gradient operators to you.

2.5 Image gradient-Sobel operator

The Sobel operator includes two matrices, one is to calculate the gradient in the horizontal direction, and the other is to calculate the gradient in the vertical direction.
Then this * operation is to calculate the dot product of the two matrices, It means multiplying the corresponding position elements in the matrix and then adding them.
And some of them are 1 and some are 2. In fact, this is the embodiment of high-number functions. We see that the operator with a value of 2 or -2 is closer to the center point

#dst = cv2.Sobel(src, ddepth, dx, dy, ksize)
# ddepth:图像的深度,处理结果图像深度，
# 一般我们都填-1，即与原图深度相同。但在这里我们需要填写cv2.CV_64F。
# 简单来说就是如果填写-1，我们在计算梯度时产生的负数就会被程序默认为0，
# 导致有一边的边缘出不来。而cv2.CV_64F范围更大，可以保留负数

# dx和dy分别表示水平和竖直方向

# ksize是Sobel算子的大小

def cv_show(img,name):
    cv2.imshow(name,img)
    cv2.waitKey()
    cv2.destroyAllWindows()
sobelx = cv2.Sobel(img,cv2.CV_64F,1,0,ksize=3)
cv_show(sobelx,'sobelx')

sobely = cv2.Sobel(img,cv2.CV_64F,0,1,ksize=3)
# 把算出来的soblex和sobley合并
sobelxy = cv2.addWeighted(sobelx,0.5,sobely,0.5,0)
cv_show(sobelxy,'sobelxy')
# 直接把x,y都计算出来，但是效果不是很好，轮廓出来是断断续续的，像是被等分成了4份
sobelxy=cv2.Sobel(img,cv2.CV_64F,1,1,ksize=3)
sobelxy = cv2.convertScaleAbs(sobelxy) 
cv_show(sobelxy,'sobelxy')

This is the original picture

# 下面我们用一个具体的例子看一下效果
img = cv2.imread('lena.jpg',cv2.IMREAD_GRAYSCALE)
sobelx = cv2.Sobel(img,cv2.CV_64F,1,0,ksize=3)
sobelx = cv2.convertScaleAbs(sobelx)
sobely = cv2.Sobel(img,cv2.CV_64F,0,1,ksize=3)
sobely = cv2.convertScaleAbs(sobely)
sobelxy = cv2.addWeighted(sobelx,0.5,sobely,0.5,0)
cv_show(sobelxy,'sobelxy')

This is the contour after passing the sobel operator

This is the morphological contour. Is it very blurry? This also shows that for complex images, the contour effect obtained by ordinary morphology is not very good, and contour operators such as sobel are still necessary

2.6 Image gradient - Scharr operator

We can see that the overall structure of the Scharr operator is similar to that of the Sobel operator. The Scharr operator enhances the difference of the Sobel operator, so the two have the same principles and usage methods for detecting image edges. The size of the edge detection filter of the Scharr operator is 3×3, so it is also called the Scharr filter. The difference between pixel values can be increased by amplifying the weight coefficient in the filter to make up for the shortcomings of the Sobel operator's poor extraction effect on weak edges in the image.

2.7 Image gradient-Laplacian operator

The Laplacian operator is the basis for pixel grayscale difference calculation in the image field. It is an image neighborhood enhancement algorithm derived through second-order differentiation. The basic idea is that when the grayscale of the central pixel in the neighborhood is lower than the average grayscale of other pixels in its neighborhood, the grayscale of the central pixel should be further reduced, and when it is higher than the average grayscale of the central pixel, the grayscale of the central pixel should be further increased.

For the specific principle, you can read the following article. By mainly understanding the derivation of discrete points, you can understand how the laplacin operator comes from.
The principle of Laplacin operator:https://www.jianshu.com/p/ca96bf5b4ad7
Let’s look at the differences below Differences in operators

#不同算子的差异
img = cv2.imread('lena.jpg',cv2.IMREAD_GRAYSCALE)
sobelx = cv2.Sobel(img,cv2.CV_64F,1,0,ksize=3)
sobely = cv2.Sobel(img,cv2.CV_64F,0,1,ksize=3)
sobelx = cv2.convertScaleAbs(sobelx)   
sobely = cv2.convertScaleAbs(sobely)  
sobelxy =  cv2.addWeighted(sobelx,0.5,sobely,0.5,0)  

scharrx = cv2.Scharr(img,cv2.CV_64F,1,0)
scharry = cv2.Scharr(img,cv2.CV_64F,0,1)
scharrx = cv2.convertScaleAbs(scharrx)   
scharry = cv2.convertScaleAbs(scharry)  
scharrxy =  cv2.addWeighted(scharrx,0.5,scharry,0.5,0) 

laplacian = cv2.Laplacian(img,cv2.CV_64F)
laplacian = cv2.convertScaleAbs(laplacian)   

res = np.hstack((sobelxy,scharrxy,laplacian))
cv_show(res,'res')

The first one is the Sobel operator
The second one is the Scharr operator: We see that the Scharr operator depicts images more carefully than the Sobel operator and can capture detailed gradients
The third one is the Laplacin operator: We can see that the laplacin operator depicts a lot of things very clearly. This is because the laplacin operator is relatively sensitive. Before using the laplacin operator, the noise points must be removed. .

2.8 Canny edge detection

step:

**
Calculate image gradient
non-maximum suppression
Dual threshold detection
Suppress isolated weak edges

Image noise reduction

We know that gradient operators can be used to enhance images and enhance the layering of images. This is essentially achieved by enhancing edge contours, which means that edges can be detected. However, they are highly affected by noise. Because noise is where the grayscale changes greatly, it is easily identified as a false edge. So we need to remove the noise points first

Calculate image gradient

After denoising, we calculate the image gradient to get a possible edge set (because the gradient is where the grayscale changes are obvious, and the edges are also where the grayscale changes are obvious.) But what we get is the possible edge, not necessarily the edge. Because the place where the grayscale changes may be the edge, or there may be many edges.

non-maximum suppression

Usually the places where the grayscale changes are relatively concentrated. On the gradient within the local range, the grayscale changes with the largest changes are retained, and the others are not retained. This can also be proposed to remove a large part and turn the edges with multiple pixels wide into A single-pixel wide edge. That is, the "fat edge" becomes the "thin edge". We mentioned above why we do not use morphological gradient calculations but operator calculations. This is also the reason. If they are retained, the outlines of the entire image will look like they are squeezed together.

Dual threshold detection

After non-maximum suppression, there are still many possible edge points, and a double threshold is further set, namely low threshold (low) and high threshold (high). If the gradient is greater than high, it is set as a strong edge pixel, and if it is lower than low, it is eliminated. Set the low and high own to weak edges. Further judge, if there is a strong edge pixel in its area, keep it, if not, eliminate it.

img=cv2.imread("lena.jpg",cv2.IMREAD_GRAYSCALE)
v1=cv2.Canny(img,80,150)
v2=cv2.Canny(img,50,100)

res = np.hstack((v1,v2))
cv_show(res,'res')

Summarize

This article mainly talks about some operations related to image processing. The image operations should be updated in the next issue. The next issue will bring a bank card number recognition project, so stay tuned.

I'm Mayphry, from a little bit to a billion points, see you next time