Chapter 7: Image Smoothing

The smoothing of the image is to filter out the noise inside the image while keeping the original information of the image as much as possible. Because image smoothing is usually accompanied by image blurring, sometimes image smoothing is also called image blurring. It is also sometimes called image filtering in computers. This is just a name, and different places may have different sayings, so don't get too entangled.

• The purpose of image smoothing is to filter out the internal noise of the image

The essence of image smoothing processing is to process the pixel values ​​in the image that are quite different from those of the surrounding pixels, and adjust its value to an approximate value of the pixel values ​​of the surrounding pixels.

as the picture shows:

The pixels located in the third row and the third column have obvious differences in value from the surrounding pixel values. Reflected on the image, the pixels around this point are all grayscale points, and the color of this point is darker, a black point may be noise, and the value of this point needs to be adjusted to an approximate value of the surrounding pixel values.

There are many ways of image smoothing processing, the main introduction is as follows:

• mean filtering
• box filtering
• Gaussian filter
• median filter
• bilateral filtering
• 2D convolution (custom filtering)

1. Mean filtering:

​ Mean filtering refers to using the average value of the N*N pixel values ​​around the current pixel point to replace the current pixel value with the current pixel point as the center. Using this method to traverse and process each pixel in the image can complete the mean filtering of the entire image.

​ For image edge pixels, you can only take the average value of the pixel values ​​​​of the surrounding neighborhood points in the image. It is also possible to expand the edge of the image, and fill in different pixel values ​​in the newly added rows and columns. On this basis, calculate the mean value of N · N neighborhood pixels for the original image. OpenCV provides a variety of boundary processing methods, and we can choose different boundary processing modes according to actual needs.

1. Function syntax:

• In OpenCV, the function that realizes the mean value is cv2.blur(), and its syntax format is:

  • dst = cv2.blur(src, ksize, anchor, borderType)

    • dst: return value, the processing result obtained after mean filtering.

    • src: The image to be processed, that is, the original image. There can be any number of channels and each channel can be processed independently. The depth of the image should be one of CV_8U, CV_16U, CV_16S, CV_32F, CV_64F.

    • ksize: is the size of the filter kernel. The filter kernel size refers to the height and width of its neighborhood image during mean processing. For example, its value may be (5, 5), indicating that the neighborhood mean value of size 5×5 is used as the result of the image mean value filtering process.

      

      M and N correspond to height and width, respectively. In general, M and N are equal, such as the more commonly used 3×3, 5×5, 7×7, etc. If the values ​​of M and N are larger, the number of pixels involved in the operation will be more, and the image distortion will be more serious.

    • Anchor: anchor point, its default value is (-1, -1), which means that the current point for calculating the mean is located at the center point of the kernel, and the default value is enough. In special cases, a different point can be specified as the anchor point .

    • borderType: border style, this value determines how to process the border, generally you do not need to consider the value of this value, just use the default.

Example:

import cv2

img = cv2.imread('../lena512color_noise.tiff')
dst1 = cv2.blur(img, (7, 7))
dst2 = cv2.blur(img, (15, 15))
cv2.imshow('img', img)
cv2.imshow('dst1', dst1)
cv2.imshow('dst2', dst2)
cv2.waitKey()
cv2.destroyAllWindows()

2. Box filtering:

​ Box filtering is different from mean filtering in that box filtering does not calculate pixel mean values. In mean filtering, the filtering result is the neighborhood average of any point. Box filtering can freely choose whether to normalize the result of mean filtering, and can freely choose whether the filtering structure is the sum of neighborhoods or the average of the sum of neighborhoods.

1. Box filter syntax:

In OpenCV, the function that implements box filtering is cv2.boxFilter(), and its syntax format is:

dst = cv2.boxFilter(src, ddepth, ksize, anchor, normalize, borderType)

• dst: return value, the processing result obtained after mean filtering.

• src: The image to be processed, that is, the original image. There can be any number of channels and each channel can be processed independently. The depth of the image should be one of CV_8U, CV_16U, CV_16S, CV_32F, CV_64F.

• ddepth: The image depth of the result image, generally use -1 to indicate the same image depth as the original image.

• ksize: is the size of the filter kernel. The filter kernel size refers to the height and width of its neighborhood image during mean processing. For example, its value may be (5, 5), indicating that the neighborhood mean value of size 5×5 is used as the result of the image mean value filtering process.

• anchor: Anchor point, the default value is (-1, -1), which means that the current calculation point is located at the center point of the kernel, the default value can be used for modification, and a different point can be specified as the anchor point in special cases.

• normalize: Indicates whether to perform normalization processing (normalization: here refers to normalizing the calculation result to a reasonable range, that is, normalizing the calculation result to a value within the current pixel value range) during filtering. This parameter is a logical value.

  • When the parameter normalize=1, it means to perform normalization processing, and the value of the neighboring pixels and the processing area should be used.
  • When the parameter normalize=0, it means that normalization processing is not required, and the sum of neighboring pixel values ​​is directly used. (255 when the sum exceeds 255)

  Usually, for box filtering, the convolution kernel can be expressed as:
  

The above corresponding relationship is:
$$f(n)=\{\begin{array}{ll}1/width\cdot height,& \text{normalize=1}\\ 1,& \text{normalize=0}\end{array}$$

• borderType: border style, this value determines how to handle the border.

Normally, when using the box filter function, for the parameters anchor, normalize, and borderType, you can directly use the default values.

Example:

import cv2

img = cv2.imread('../lena512color_noise.tiff')
rst = cv2.boxFilter(img, -1, (7, 7))
cv2.imshow('img', img)
cv2.imshow('rst', rst)
cv2.waitKey()
cv2.destroyAllWindows()

3. Gaussian filter:

In mean filtering and box filtering, the weight of each pixel in its neighborhood is equal. In Gaussian filtering, the weight closer to the center point is increased, and the weight farther away from the center point is decreased. On this basis, the sum of different weights of each pixel value in the neighborhood is calculated.

1. Basic principles

• In Gaussian filtering, the value of the convolution kernel is not all 1. as shown in the picture

  

• In Gaussian filtering, the width and height of the convolution kernel can be different, but they must be odd numbers.

  

• Each size of convolution kernel can have various weight ratios in different forms. As shown in the figure, it is also a 5x5 convolution kernel.

  

  In different materials, the convolution kernel has many different representations. They may be written in a table, or they may be shown in a matrix. In actual calculation, the convolution kernel is normalized, which can be expressed as a convolution kernel in decimal form or in fractional form. The convolution kernel without normalization is used to illustrate the problem. Strictly speaking, using a convolution kernel that has not been normalized for filtering, the results are often wrong.

2. Function syntax :

In OpenCV, the function cv2.GaussianBlur() that implements Gaussian filtering, the syntax format of this function is:

dst = cv2.GauusianBlur(src, ksize, sigmaX, sigmaY, borderType)

• dst: return value, the processing result obtained after mean filtering.

• src: The image to be processed, that is, the original image. There can be any number of channels and each channel can be processed independently. The depth of the image should be one of CV_8U, CV_16U, CV_16S, CV_32F, CV_64F.

• ksize: is the size of the filter kernel. The filter kernel size refers to the height and width of its neighborhood image during mean processing. However, it should be noted that the value of the filter kernel must be an odd number.

• sigmaX: The standard deviation of the convolution kernel in the horizontal direction (X-axis direction), which controls the weight ratio.

• sigmaY: The standard deviation of the convolution kernel in the vertical direction (Y-axis direction). If this value is set to 0, only the value of sigmaX will be used; if both sigmaX and sigmaY are 0, pass ksize.width and ksize.height Calculated to get:

  • sigmaX = 0.3 × [(ksize.width-1) × 0.5-1] + 0.8
  • sigmaY = 0.3 × [(ksize.height-1) × 0.5-1] + 0.8

• borderType: border style, this value determines how to handle the border. Under normal circumstances, this value does not need to be considered, and the default value is used directly.

Note: sigmaY and borderType are optional parameters in this function. sigmaX is a mandatory parameter, but it can be set to 0 to let the function calculate the specific value of sigmaX by itself.

​ The official document recommends specifying the values ​​of the three parameters ksize, sigmaX, and sigmaY to avoid syntax errors that may be caused by future parameter modifications. Of course, in actual processing, the specified sigmaX and sigmaY that can be displayed are the default values ​​of 0. Therefore, the common form of the function cv2.GaussianBlur() is:

• dst = cv2.GaussianBlur(src, ksize, 0, 0)

import cv2
o=cv2.imread（"image\\lenaNoise.png"）
r=cv2.GaussianBlur（o,（5,5）,0,0）
cv2.imshow（"original",o）
cv2.imshow（"result",r）
cv2.waitKey（）
cv2.destroyAllWindows（）

4. Median filtering

​ Median filtering is different from the previous filtering method, and the filter value is no longer used to calculate the filtering result. It uses the median value of all pixel values ​​in the neighborhood to replace the pixel value of the current pixel.

1. Basic principles:

​ Median filtering will take the pixel values ​​of the current pixel and its surrounding adjacent pixels (a total of odd pixels), sort them, and then use the pixel value in the middle position as the pixel value of the current pixel.

2. Function syntax:

In OpenCV, the function that implements median filtering is cv2.medianBlur(), and its syntax format is:

dst = cv2.medianBlur(src, ksize)

• dst: return value, the processing result obtained after mean filtering.

• src: The image to be processed, that is, the original image. There can be any number of channels and each channel can be processed independently. The depth of the image should be one of CV_8U, CV_16U, CV_16S, CV_32F, CV_64F.

• ksize: is the size of the filter kernel. The filter kernel size refers to the height and width of its neighborhood image during mean processing. But it should be noted that the value of the filter kernel must be an odd number greater than 1, such as 3, 5, 7, etc.

Example:

import cv2

img = cv2.imread('../lena512color_noise.tiff')
rst = cv2.medianBlur(img, 5)
cv2.imshow('img', img)
cv2.imshow('rst', rst)
cv2.waitKey()
cv2.destroyAllWindows()

​ It can be seen from the results that since no mean value processing is performed, the median filter does not have the problem of fuzzy details caused by filtering methods such as mean filtering. In median filter processing, it is difficult to select noise components, so it is possible to extract all noise without affecting the original image. But the disadvantage is that due to the need for operations such as sorting, the median filter requires a large amount of calculations.

Five, bilateral filtering:

Bilateral filtering is a filtering method that comprehensively considers spatial information and color information, and can effectively protect the edge information in the image during the filtering process.

1. Basic principles:

The previous filtering methods only consider the spatial weight information, which is more convenient to calculate, but there will be a big problem when processing edge information. For example, the left side of the image is black, the right side is white, and there is a sharp edge in the middle.

In mean filtering, box filtering, and Gaussian filtering, the weighted average of each pixel on the edge is calculated, thereby blurring the edge information. The following figure is the image processed by Gaussian filtering:

When calculating the new value of a pixel, bilateral filtering not only considers the distance information (the farther the distance, the smaller the weight) but also considers the color information (the greater the color difference, the smaller the weight). Bilateral filtering considers the weight structure of distance and color comprehensively, which can not only effectively remove noise, but also better protect edge information.

​ In bilateral filtering, when it is on the edge, pixels that are similar in color to the current point (the color distance is very close) will be given a larger weight value; while pixels that are more different from the current color (the color distance is far ) will be given a smaller weight value (the weight may be 0 in extreme cases, just ignore this point), thus protecting the edge information.

2. Function syntax:

In OpenCV, the function that implements bilateral filtering is cv2.bilateralFilter(). The syntax of this function is:

dst = cv2.bilateralFilter(src, d, sigmaColor, sigmaSpace, borderType)

• dst: return value, the processing result obtained after mean filtering.
• src: The image to be processed, that is, the original image. There can be any number of channels and each channel can be processed independently. The depth of the image should be one of CV_8U, CV_16U, CV_16S, CV_32F, CV_64F.
• d: The space distance parameter selected during filtering, here represents the diameter of the current center point. If the value is non-positive, it will be automatically calculated from the parameter sigmaSpace. If the filtering space is large (d>5), the speed is slower. Therefore, d=5 is recommended in practical applications. For off-line filtering with larger noise, d=9 can be selected.
• sigmaColor: The color difference range selected during filtering, which determines which surrounding pixels can participate in filtering. Pixels whose pixel value difference with the current pixel is smaller than sigmaColor can participate in the current filtering. The larger the value, the more surrounding pixels can participate in the calculation. When the value is 0, the filtering is meaningless; when the value is 255, all points within the specified diameter can participate in the calculation.
• sigmaSpagce: is the sigma value in the coordinate space. The larger its value, the more points can participate in the filtering calculation. When d > 0, d specifies the neighborhood size regardless of the value of sigmaSpace; otherwise, d is proportional to the value of sigmaSpace.
• borderType: border style, this value determines how to handle the border. Under normal circumstances, this value does not need to be considered, and the default value is used directly.

​ For simplicity, you can set the values ​​of sigmaColor and sigmaSpagce to be the same. If their values ​​are relatively small (less than 10), the filtering effect is not obvious; if their values ​​are relatively large (for example: greater than 150), the filtering effect will be more obvious, and a cartoon effect will be produced.

In the cv2.lateralFilter() function, except borderType is an optional parameter, all other parameters are required.

import cv2

img = cv2.imread('../lena512color_noise.tiff')
rst = cv2.bilateralFilter(img, 25, 100, 100)
cv2.imshow('img', img)
cv2.imshow('rst', rst)
cv2.waitKey()
cv2.destroyAllWindows()

It can be seen from the above results that bilateral filtering is not effective in removing noise. The advantage of bilateral filtering is reflected in the processing of edge information.

import cv2
import numpy as np

img = np.zeros((500, 500), dtype=np.uint8)
img[:, :250] = 255
rst = cv2.bilateralFilter(img, 25, 100, 100)
cv2.imshow('img', img)
cv2.imshow('rst', rst)
cv2.waitKey()
cv2.destroyAllWindows()

Six, 2D convolution

​ OpenCV provides a variety of filtering methods to achieve the effect of smoothing images, such as mean filtering, box filtering, Gaussian filtering, median filtering, etc. The convolution kernels used in most filtering methods have certain flexibility, and the size and number of convolution kernels can be easily set. However, sometimes we want to use a specific convolution kernel to implement convolution operations, for example, use the following convolution kernels to perform convolution operations.

None of the filter functions introduced above can determine the convolution kernel as the above form, so OpenCV's custom convolution function should be used.

​ In OpenCV, users are allowed to customize the convolution kernel to realize the convolution operation. The function of using the custom convolution kernel to realize the convolution operation is cv2.filer2D(), and its syntax format is:

dst = cv2.filter2D(src, ddepth, kernel, anchor, delta, borderType)

• dst: return value, the processing result obtained after mean filtering.

• src: The image to be processed, that is, the original image. There can be any number of channels and each channel can be processed independently. The depth of the image should be one of CV_8U, CV_16U, CV_16S, CV_32F, CV_64F.

• ddepth: The image depth of the result image, generally use -1 to indicate the same image depth as the original image.

• kenel: The convolution kernel is a single-channel array. If you want to use different kernels for each channel when processing a color image, you must decompose the color image and use different kernels to operate.

• anchor: Anchor point, the default value is (-1, -1), which means that the current calculation point is located at the center point of the kernel, the default value can be used for modification, and a different point can be specified as the anchor point in special cases.

• delta: correction value. Optional, if this value exists, it will be added to the basic filtering result as the final filtering result.

• borderType: border style, this value determines how to handle the border. Under normal circumstances, this value does not need to be considered, and the default value is used directly.

Normally, when using the filter function cv2.filter2D(), you can directly use the default values ​​for the parameters anchor, delta, and borderType.

Example: Customize a convolution kernel, apply the convolution kernel to filter the image through the function cv2.filter2D(), and display the filtering result.

import cv2
import numpy as np

img = cv2.imread('../lena512color_noise.tiff')
kernal = np.ones((9, 9), dtype=np.float32) / 81
rst = cv2.filter2D(img, -1, kernal)
cv2.imshow('img', img)
cv2.imshow('rst', rst)
cv2.waitKey()
cv2.destroyAllWindows()