[Chapter 2: Basic introduction and application of OpenCv algorithm]

1. Directory

  The theme of this sharing is: the basic introduction and application of the OpenCv algorithm. Next, we will focus on the following five aspects.

2. Image processing process

  In the image processing process, it can be simply summarized as preprocessing, segmentation, morphological operations, and target frame selection.

1）Usually, due to the influence of visual sensor material properties, working environment, electronic components, etc., images often have noise and uneven illumination, disrupting the observable information of the image. Therefore, our first step is to perform pre-processing operations such as noise reduction, filtering, and enhancement on the image to make it more suitable for subsequent processing.
2）The second step of image segmentation is also the most critical step in the processing process. The main purpose of image processing is to segment and select the target area of ​​interest. That is to separate the target from the background. Among them, the classic traditional segmentation methods include threshold segmentation, edge segmentation, etc., and deep learning includes semantic segmentation and instance segmentation.
3）In the third step, after the target and background segmentation is completed, it is often necessary to perform operations related to mathematical operations between images such as morphology, and perform operations such as removing noise points, filling holes, and segmenting adhesive objects on the segmented binary images.
4）In order to facilitate observation and display, the last step is to select the target.

3. Image preprocessing

  Image preprocessing mainly includes image enhancement and some grayscale, normalization, scale transformation, cropping, image translation, rotation and other operations. The main content shared in this preprocessing is the image enhancement part. The purpose of image enhancement is to optimize the image quality and facilitate subsequent algorithm processing. The following will cover noise classification, how to remove noise and sharpen edges, and how to adjust when the image illumination is uneven.

3.1 Noise classification

3.1.1 Random noise

  For noise, its classification is mainly related to the characteristics and distribution characteristics of the noise. Here are three common types of noise. The first is random noise. The noise distribution does not have any regular characteristics. The cancellation effect is better with mean filtering.

3.1.2 Salt and pepper noise

   The second type is salt and pepper noise, which presents a kind of abnormal pixels with maximum and minimum values, that is, black 0 white 255 pixels. Median filtering is better at removing it.

3.1.3 Gaussian noise

  The third type is Gaussian noise, whose distribution conforms to the Gaussian distribution. The Gaussian distribution is the normal distribution that we are familiar with, and its removal effect is better with Gaussian filtering. Of course, there are many other factors involved in noise, so I won’t introduce them too much here.

3.2 Filtering

3.2.1 Mean filtering

After introducing the types of noise, let's introduce how to filter images according to different noises to obtain higher quality images.
1）The first is mean filtering, whose principle is to replace the value of each pixel with the average value of the pixels in the field containing it.
2）The filter kernel given in the second picture is 3X3 in size, and the red middle point is replaced by the average value of the pixels in the 3x3 field containing it. After traversing the image, you can get a blurred image.
3）The third picture shows the filtering effects of different filters on random noise, among which the mean filter has a good effect. In fact, we can also see that the effect of median filtering is also good. Therefore, when denoising, it is not necessary that only mean filtering can be selected for random noise. The selected filter is not fixed and needs to be adjusted according to the actual effect.

3.2.2 Median filtering

• 1) The second is median filtering. Its principle is similar to mean filtering. The previous one is the mean value instead of the central value of the filter window. The median value is used here.

• 2) As shown in the first picture, sort all pixels relative to the filter window to obtain the median value and replace the pixel value at the center point of the sliding window. Here, 255 becomes 10 after filtering.

• 3) In the second picture, mean filter, median filter, and Gaussian filter are used to filter the image containing salt and pepper noise, among which the median filter has a better effect.

  

3.2.3 Gaussian filtering

1）The third type is Gaussian filtering. The principle is that the weight of each pixel is distributed according to Gaussian. The closer it is to the center point of the window, the greater the full-time, and the greater the impact on the central pixel.
2）Here I generated 3x3 and 15x15 Gaussian filter sums and visualized them. The closer to the center, the larger the pixel value and the higher the brightness. The filtering principle is similar to the previous two, and the three are all different in filter kernel.
3）Through the experiment in Figure 3, the Gaussian noise is filtered by mean, median, and Gaussian filtering, and the effect of Gaussian filtering is relatively good.

3.2.4 Bilateral filtering

1）The first three filters will also weaken the edge information while removing the impact of noise on the image. Therefore, bilateral filtering is briefly mentioned here. Bilateral filtering is edge preserving filtering. It can blur the image and reduce the impact of noise while also retaining the edge information. edge information. '
2）The principle is: based on Gaussian filtering, a folding process that combines the spatial proximity of the image and the similarity of the pixel values. In other words, the spatial proximity and the similarity of the pixel values ​​are the distance between the pixels and The value of the pixel. When the distance between two pixels is very close, and when the color is very close, the impact will be greater. On the contrary, although the distance is very close, the color difference is large, that is, the edge, then the smoothing weight will be very small. This ensures that the pixel values ​​near the edges are maintained and achieves an edge-preserving effect.
3）As can be seen from the above two pictures, simply applying Gaussian blur to the image blurs various edge information, while bilateral filtering retains edge information on the basis of blurring.

3.2.4 Motion blur

1）The previous filtering is mainly to reduce the impact of noise on image quality. It is considered that filter blur is introduced. The motion blur introduced below is due to the slow shutter speed of the camera or the fast moving speed of the object, as well as the vibration or movement during the shooting process, resulting in the captured image. A slight displacement occurs, causing the image to blur in one direction.
2）The solutions include physical methods such as increasing the shutter speed, increasing the camera frame rate, increasing damping and buffering vibration, and there are also software methods that use image repair networks to de-blur.

3.2.4 Sharpening filter

1）The previous article mainly talked about noise and how to remove noise and blur. The following is image sharpening, which is the opposite of blur.
2）The principle is to enhance the high-frequency information in the image to make the image details clearer. The so-called high-frequency information refers to the rapidly changing parts of the image, such as image edges, corners, etc. Through sharpening filtering, slide through the image, and finally output the sharpened image.
3）In order to facilitate observation, as shown in Figure 1, input an image, and the pixel value of the contour 1 is 1-10. After filtering, the difference between the pixel value of the contour 1 and the adjacent pixel value is larger, the contrast is stronger, and it can better display its The outline edges of the image.
4）Among them, image sharpening is widely used in surveillance and security. There are pictures without sharpening and dynamic pictures with sharpening added. Compared with the original picture, it is easier to see the target clearly in the sharpened image. The sum of the filter kernel values ​​will also affect the brightness of the image. If it is greater than 1, the image will increase the variable. Here, if the sum is set to 0, the image will become darker and become edge detection.

3.3 Image enhancement

3.3.1 Histogram equalization

1）In image preprocessing, in addition to denoising blur and edge sharpening, because the images we obtain sometimes result in excessive or too dark images due to uneven lighting, we often need to equalize the image and adjust the overall brightness of the image. and contrast. Here we take histogram equalization as an example.
2）Image histogram, which counts the number of each pixel value. When the pixel interval geometry is within a certain interval, it proves that the picture is too bright or too dark. Therefore, some linear and nonlinear transformations can be used to evenly distribute the pixels between 0-255, thereby achieving brightness adjustment.
3）In addition, there are many ways to adjust image brightness, as shown in Figure 1, so I won’t go into details here.

经过图像预处理，下面我们将快速对图像分割板块进行介绍。图像分割图像处理中最关键的环节，下面将从阈值分割、边缘检测分割、粘连物体分割三个方面进行介绍。

4. Image segmentation

• Traditional segmentation methods mainly include threshold segmentation, edge detection segmentation, and adhesive object segmentation.
  

4.1 Classification of segmentation types

1）Image segmentation includes traditional algorithm segmentation and deep learning segmentation. Here we mainly introduce the traditional algorithm segmentation. Segmentation personally feels that it can be divided into two categories, one is the segmentation of the target and the background, and the other is the segmentation of images based on the former to segment connected objects.
2）For the segmentation of front and rear scenes, it is divided into threshold segmentation and edge detection segmentation. Threshold segmentation can be subdivided into single threshold segmentation and multi-threshold segmentation. Edge detection segmentation mainly filters the image based on differential, gradient and other related operators to obtain the contour of the image.
3）For the segmentation of adherent objects, there are three most common methods, namely morphological segmentation, watershed segmentation, and geometric segmentation. Below I will also share and introduce some examples that I have done before.

4.2 Single threshold segmentation

  The first is single-threshold segmentation in threshold segmentation. It is not only suitable for grayscale image segmentation. As long as it is a single-channel image, it can be segmented by this method. For single-threshold segmentation, you can generally use a grayscale histogram to display the distribution of image pixels. As shown in Figure 2, the background color is black and the number is large. All have a peak value near 0. The foreground is gray-white and the number is small. Set Between 130-230. Therefore, we only need to set a threshold between the two peaks to segment the front and rear backgrounds.

4.3 Separate various color space channels

  For a single channel, here is a brief insertion. When doing threshold segmentation, you cannot just limit it to the grayscale channel. Sometimes you can segment the pictures in each color space and observe which channel features are more obvious, and then use the corresponding channel to perform segmentation. In addition, color segmentation is also performed using the correlation of certain channels.

4.4 Multi-threshold segmentation

1）Compared with single-threshold segmentation and multi-threshold segmentation, it is relatively flexible. The applicable object can be single-channel multi-threshold segmentation or multi-channel multi-threshold segmentation.
2）For single-channel multi-threshold segmentation, Otsu algorithm and adaptive threshold segmentation algorithm are introduced here. The Otsu algorithm is also called the maximum inter-class difference method. The principle is to divide the gray value into multiple thresholds, combine each threshold, and divide the image into several classes. Based on the number of pixels and the average gray value of each class, find The inter-class variance under this threshold is obtained. The larger the inter-class variance, the better the segmentation effect.
3）For adaptive threshold segmentation, also known as local thresholding, the picture is first divided into several areas. The threshold of each pixel in the area is determined by the pixel value distribution of the part. The image pixel threshold in the area with higher brightness is higher, and the threshold of the image with higher brightness is higher. For lower image areas, the threshold will become smaller accordingly. Very suitable for segmentation of images with uneven lighting.
4）The above content is mainly about multi-threshold segmentation under single channel. The content shared below is multi-threshold segmentation under multi-channel. Multi-channel multi-threshold segmentation can actually be understood as performing single-threshold segmentation on each channel, and finally calculating the union of each channel. Our common color segmentation is HSV color space. This method can be appropriately extended to segmentation in other color spaces. For the way, the RGB is used to convert to HSV color space, and then the colors are segmented and detected and positioned.

4.5 Edge detection

4.5.1 Edge detection case

1）After talking about threshold segmentation, the following content is edge detection segmentation. Edge detection is also widely used in image detection.
2）Here is an example of how to segment text information and change the background when the lighting is slightly uneven. If you solve it from threshold segmentation, the effect is not so good, but using the Canny edge detection algorithm, you can avoid the impact of uneven lighting, directly detect the outline, perform morphological operations and filling on it, and then perform mask operation, you can achieve Bottom changing function.
3）In fact, edge detection is similar to the previous fuzzy filtering operation. It performs interactive convolution operations on the image through various operator templates, thereby detecting areas in the image with significant changes in pixel values, that is, edges.

4.5.3 Various edge detection operators

• Shown here are various edge detection operator templates, specific principles and corresponding advantages and disadvantages. I will not go into details here.
  

4.6 Other segmentation algorithms—background subtraction model

• In addition to the above classic segmentation methods, there is also a background subtraction algorithm that is more effective in segmenting moving objects. The first animation is the identification of passion fruit on the passion fruit sorting machine, and the second animation is the application of enemy vehicle identification and detection on the Mecha Master radar station. Due to limited time, I will speed up sharing Jiangdu later, and the specific principles will be discussed in private later.

  

DJI Mecha Master Radar Station Local Vehicle Detection

4.7 Watershed segmentation algorithm, geometric segmentation algorithm, morphological segmentation algorithm

1）The above threshold segmentation, edge detection, and background subtraction segmentation methods all belong to foreground and foreground segmentation. Sometimes the segmented objects are adherent, and in this case, an inter-class segmentation is required.
2）At present, the main divisions are watershed algorithm segmentation, geometric segmentation, and morphological segmentation. Morphological segmentation will be mentioned in the next chapter. Watershed segmentation has a relatively good effect on segmenting adhering circular objects, but if the adhesion is too severe, it is difficult to segment.
3）For geometric segmentation, I think the effect will be better in industrial inspection because the size of the parts is fixed. For example, when detecting coins on the way, I calculate the distance by detecting the focus, filtering the corner points, and filtering out the focus points of the connected depressions to obtain ROI analysis, and finally segmentation.
4）To segment the mask belt and non-woven fabric, the minimum circumscribed rectangle of the non-woven fabric is obtained, the coordinate equation is obtained based on the coordinates of the short side, and the straight line is used for cutting.

图像预处理、并分割之后，往往需要对其二值化图像进行形态学操作。

5. Morphological operations

• Morphological operations mainly include operations such as expansion, erosion, opening and closing operations, top and bottom caps, morphological gradients, and flooding filling.
  

5.1 Principle

• 1) In order to facilitate everyone to quickly understand the principle of morphological operations, an animation is created here. The principle is the same as the previous filtering operation. The structural elements slide and traverse in the input image. When the center point of the structural element is in contact with the white pixel point of the input image When overlapping, pixels that overlap with the white pixels of structural elements will turn white. Everyone can watch the animation.

• 2) The picture is a verification using the opencv API.

• 3) Here is a little trick, which is to define the structural elements. The shape can be drawn in the form of a picture. For example, I define a single-channel sunspot background image and draw a circle on it. In fact, the structural element is a circle. Other shapes can be defined directly or combined.
  

  

5.2 Expansion, corrosion, opening operation, closing operation

• The various operations of morphology will be quickly introduced below.
  1）Expansion and corrosion are relatively intuitive, that is, the white area becomes fatter and thinner.
  2）The open operation is to corrode first and then expand, which can effectively eliminate some white noise. The closing operation is the opposite. It expands first and then corrodes, which can effectively fill the void.
  

• For top hat and bottom hat, it is actually a subtraction operation between binary images. The top hat is the original image minus the opening operation, which can find the noise or the brighter areas of the image. The bottom hat subtracts the closed operation from the original color image to find the holes and separate the darker areas of the image.

5.3 Top (ceremony) hat, bottom (black) hat

5.4 Morphological gradient,

  The morphological gradient is expansion minus erosion, which can be similar to a large circle minus a small circle to obtain a concentric circle. There is also a flood filling, which randomly generates a seed pixel to generate horizontal and vertical lines and fills the outline. In fact, this operation can also achieve the same effect by using outlines.

通过图像预处理、图像分割、形态学等相关操作，最后就是为了对我们的一个目标进行框选显示保存。

6. Target frame selection

• In the target frame selection display, you can use points to display the contour centroid, or you can detect it through contours, or Hough straight lines, Hough circles, and also display right-connected rectangles, minimum circumscribed rectangles, and cross-shaped polygons.
  

  

7. Summary

• The above is the general content I shared today. I will share more content tonight, because in fact, each algorithm alone is enough to talk about for a long time, so I only briefly mention its basic ideas. I hope that my summary of my previous study of opencv can be helpful to everyone. At the same time, there may be some shortcomings due to my limited ability and some expression errors. Please correct me and discuss together. Thank you all for listening.

  

8. Remarks

Some picture materials are from the Internet and are only used for academic sharing. If there is any infringement, please contact the backend for deletion.

---

1.理论系列:

Chapter 1: Summary of environment configurations such as pycharm, anaconda, opencv, pytorch, tensorflow, paddlex, etc. [Image processing py version] ==> Chapter 2:

Basic introduction and application of OpenCv algorithm

Chapter 3: OpenCv image and video reading and writing Operation and basic application

Chapter 4: Summary of OpenCv threshold segmentation/binarization (single-channel, multi-channel images)

---

2.项目系列:

Project 1: Grade 4 and 6 Rewriting System
Project 2: Practical Combat: Adhesive Object Segmentation - Bottle Cap Segmentation Detection Using Geometric Segmentation
Project 3: Practical Combat: Adhesive Object Segmentation - Coin Segmentation Detection Using Geometric Segmentation
Project 4: Practical Combat Chapter: Segmentation of Adhesive Objects - Using Geometric Segmentation to Realize Cell Segmentation and Detection
Project Five: Practical Chapter: Segmentation of Adhesive Objects - Using Watershed Algorithm to Realize Segmentation and Detection of Sugar Beans