1. Description
In the digital age, images are an integral part of our communication and expression. From social media to medical imaging, the quality and content of images is important. This is where the fields of image filtering and convolution step in, giving us a toolkit for transforming and refining these visual narratives.
Image filtering isn't just about making a photo look better; it's about making the image tell its story. Whether reducing noise in medical scans, sharpening the intricate details of archaeological artifacts, or detecting edges in self-driving cars, image filters play a key role. They shape the narrative by emphasizing certain characteristics while subduing others.
The backbone of this process is the convolution operation, a mathematical marvel that allows us to overlay images with predefined filters to achieve different effects.
2. The concept of convolution
Convolution is a basic mathematical operation that plays an important role in image processing. It is a cornerstone technology for applying filters to images, enabling tasks such as blurring, sharpening, and edge detection.
The core of convolution is to combine two functions to produce a third function. In image processing, one of the functions is the input image and the other is a smaller matrix called a kernel or filter. The kernel contains numerical values that define the behavior of the filter. The process involves sliding a kernel over the image, multiplying its value with the corresponding pixel value in the image, and then summing the results. This summation is then placed at a specific location in the output image.
2.1 One-dimensional convolution
In the context of 1D convolutions, our input consists of 1D arrays, often called feature vectors.
In the most basic convolution scenario, we perform a multiplication between this array and numerical values (called filters or weights) to produce an output array called a feature map.
In this case, our initial array has dimension 12x1 and the filter has dimension 1x1, therefore, the generated output remains 12x1 in size.
Additionally, shown below is a filter of size 2x1.
Here we multiply the first element of the input by the first element of the filter, the second element of the input by the second element of the filter and then add them together and write the result to the output array in the first element.
Here, as we can observe, we cannot move the filter further; therefore, the output will be smaller, with size 11x1.
2.2 Filling
Padding is the practice of adding extra pixels around the edges of an image. The purpose of padding is to affect the size of the output image after applying a convolution operation using a specific kernel or filter.
Below, we add zeros to both ends of the input array, so we end up with the same size output.
So far we've moved the filters one by one to the right, but we don't necessarily have to keep doing this; for example, we could move it three.
2.3 One-dimensional convolution
Now we have an 8-bit grayscale image with a 5x5 grid. The values range between 0 and 255.
And we have a random 3x3 kernel:
When we apply the kernel to the input:
（-1*170）+（0*245）+（1*0）+（2*234）+（1*42）+（2*64）+（1*32）+（-2*53）+（0*128）=394
We replace the center pixel with the calculated value. We then move the kernels one by one (from top left to bottom right) and do the math again.
Again we use padding for the edge pixels.
3. Filter
The image filter's versatility is achieved through its series of operations. Each operation has a different purpose, shaping the beauty of the image or facilitating deeper analysis.
- smoothing filter
- sharpening filter
- edge enhancement filter
3.1 Smoothing filter
Smoothing filters, also known as blur filters, have the remarkable ability to improve image quality by gently reducing noise, suppressing unwanted detail, and creating a more harmonious visual experience.
Essentially, a smoothing filter works by averaging the pixel value of an image with its neighboring pixels. This averaging process has a soothing effect on the image, making it more cohesive and even. The resulting image retains its main features but cleans up the minor fluctuations and inconsistencies that might obscure its essence.
- box
- Gaussian
- median number
3.2 Frame blur
This filter performs a simple average of pixel values within a defined neighborhood. The size of the neighborhood is called the kernel size . The larger the kernel size, the blurrier the image.
It is a simple and efficient smoothing filter. It is often used as the first step in image processing algorithms, such as edge detection and noise reduction.
import cv2
import numpy as np
import matplotlib.pyplot as plt
path = "cath.jpeg"
# read image
image = cv2.imread(path)
# box blur filter
box_blur = cv2.boxFilter(image, -1, (25, 25))
# plot
plt.figure(figsize=(10, 8))
plt.subplot(2, 2, 1)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 2, 2)
plt.imshow(cv2.cvtColor(box_blur, cv2.COLOR_BGR2RGB))
plt.title('Box Blur')
plt.axis('off')
The frame is blurry. Image provided by the author.
The method we use results in a simple averaging filter.boxFiltercv2
cv.boxFilter（src， ddepth， ksize[， dst[， anchor[， normalize[， borderType]]]]） -> dst
srcis the input image.ddepthis the desired depth of the output image. Set it to -1 to make the output image have the same depth as the input image. Otherwise, you can use , , .cv2.CV_8Ucv2.CV_16Ucv2.CV_32Fcv2.CV_64Fksizeis the size of the box core. The larger the kernel, the stronger the smoothing effect.dstis optional and outputs an image. If not provided, the function will create a new output image.anchoris an optional anchor point in the kernel. It is usually set to represent the center of the kernel. Changing this setting may affect the position of the filtered output.-1, -1normalizeis an optional flag indicating whether the kernel should normalize by the number of elements in it. Normalization helps keep the brightness of the image relatively constant.borderTypeIs an optional flag indicating how to handle image borders.
3.3 Gaussian filter
It takes a weighted average of pixel values, with weights determined by a Gaussian distribution. The weight of neighboring pixels decreases as they move away from the central pixel.
It is a more complex smoothing filter than box blur. It better preserves edges and details in images while reducing noise.
gaussian_blur = cv2.GaussianBlur(image, (25, 25), 0)
plt.figure(figsize=(10, 8))
plt.subplot(2, 2, 1)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 2, 2)
plt.imshow(cv2.cvtColor(gaussian_blur, cv2.COLOR_BGR2RGB))
plt.title('Gaussian Blur')
plt.axis('off')
Gaussian blur. Image provided by the author.
cv.GaussianBlur（src， ksize， sigmaX[， dst[， sigmaY[， borderType]]]） -> dst
sigmaXis the standard deviation of the Gaussian kernel in the horizontal direction. The larger the value, the stronger the blur effect.sigmaYis optional, the standard deviation of the Gaussian kernel in the vertical direction. If not specified, defaults to the same value as .sigmaX
3.4 Median filter
The median filter replaces each pixel value in the image with the median value of its neighboring pixels. The median is the middle value in a sorted list of values. This means that the median filter is not affected by outliers such as noise spikes.
median_filtered = cv2.medianBlur(image, 25)
plt.figure(figsize=(10, 8))
plt.subplot(2, 2, 1)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
plt.title( 'Original Image' )
plt.axis( 'off' )
plt.subplot(2, 2, 2)
plt.imshow(cv2.cvtColor(median_filtered, cv2.COLOR_BGR2RGB))
plt.title( 'Median Filter' )
plt.axis( 'off' )
Median filter. Image provided by the author.
cv.medianBlur（src， ksize[， dst]） -> dst
4. Sharpening filter
Sharpening filters are a type of image processing technique used to enhance edges and fine details in images. Unlike blur filters, which smooth image details, sharpen filters highlight differences in pixel intensity between adjacent areas, thereby enhancing edges and making features more prominent.
The sharpening filter works by enhancing the high-frequency components of an image. High-frequency components correspond to rapid changes in pixel values, which are characteristic of edges and fine details.
Sharpening filters can produce visually appealing effects, but should be used with caution. Over-sharpening can amplify noise and introduce artifacts. It's often good practice to apply a sharpening filter to an image that's already of good quality and has minimal noise.
- Laplacian filter
- Unsharp mask
4.1 Laplacian filter
The Laplacian filter is an edge detection filter used in image processing. It is a second derivative filter, which means it measures the rate of change of image intensity. Laplacian filters are commonly used to sharpen images and detect edges.
laplacian = cv2.Laplacian(image, -1, ksize=5, scale=1,delta=0,borderType=cv2.BORDER_DEFAULT)
plt.figure(figsize=(10, 8))
plt.subplot(2, 2, 1)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 2, 2)
plt.imshow(laplacian)
plt.title('Laplacian Filter')
plt.axis('off')
Laplacian filter. Image provided by the author.
cv.Laplacian（src， ddepth[， dst[， ksize[， scale[， delta[， borderType]]]]]） -> dst
ksizeis optional, the kernel size to use for Laplacian operations. It is set to an odd number, such as,,, etc. If not provided, the default value is, which corresponds to a 3x3 core.1351scaleis optional and is the scaling factor applied to the calculated Laplacian value. This parameter helps to enhance or reduce the effect of the Laplacian.deltais optional and is a value to add to the calculated Laplacian value. It can be used to control the overall brightness or contrast of the output.
4.2 Unsharp mask
It involves creating a blurred version of the original image (using a low-pass filter) and then subtracting this blurred version from the original image. The result is an image with highlighted edges and details. The term "unsharp" refers to creating a mask that highlights the unsharp content in the image.
The amount of sharpening is controlled by the amount of blur and the weight of the blurred image. The blur is usually a Gaussian blur, but other types of blur can also be used. Blurred images usually have weights between 0 and 1. Higher weight will result in more sharpening.
blurred = cv2.GaussianBlur(image, (25, 25), 3)
k = 3
unsharp_mask = cv2.addWeighted(image, k-1, blurred, -1.0, 0)
plt.figure(figsize=(10, 8))
plt.subplot(2, 2, 1)
plt.imshow(cv2.cvtColor(image, cv2.COLOR_BGR2RGB))
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 2, 2)
plt.imshow(cv2.cvtColor(unsharp_mask, cv2.COLOR_BGR2RGB))
plt.title('Unsharp Masking')
plt.axis('off')
Unsharp masking. Image provided by the author.
High boost filtering is a common form of unsharp masking. It allows you to control the amount of enhancement applied to image details. It does this by adding a weighted combination of the original image and a blurred version of the image, where the weight determines the degree of enhancement.
This function is used to combine two images with different weights. In this context, it is used to perform high-boost filtering operations.cv2.addWeighted
image: This is the original image.k - 1: This is the weight of the original image. Subtracting 1 from this ensures that the contribution of the original image is scaled according to the enhancement factor. Value puts more emphasis on enhanced detail.kk - 1blurred: This is a blurry image.-0.5: This is the weight of the blurred image. Negative values subtract blur from the image.0: This is an offset value. After weighted addition, it is added to the result.
5. Edge enhancement filter
The edge enhancement filter is an image filter used to enhance edges in an image. Edges are the boundaries between different regions in an image, and they are important for providing information about the structure and shape of objects in the image.
- Sobel filter
- Prewett filter
- Gaussian filtered Laplacian
- Smart edge detector
5.1 Sobel filter
The Sobel filter is used to detect edges by calculating the gradient of the image intensity at each pixel. It uses two separate convolution kernels (one for the horizontal direction and one for the vertical direction) to approximate the gradient.
Sobel Operator. source
The Sobel filter is a popular edge detection filter because it is relatively simple to implement and very effective at detecting edges. However, it can also be sensitive to noise, so it is often used in conjunction with other filters such as Gaussian Blur to reduce noise in the image before applying the Sobel filter.
image = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
# Apply Sobel Filter
sobel_x = cv2.Sobel(image, cv2.CV_64F, 1, 0, ksize=25)
sobel_y = cv2.Sobel(image, cv2.CV_64F, 1, 0, ksize=25)
sobel_edges = np.sqrt(sobel_x**2 + sobel_y**2)
# plot
plt.figure(figsize=(12, 8))
plt.subplot(2, 3, 1)
plt.imshow(image, cmap='gray')
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 3, 2)
plt.imshow(sobel_edges, cmap='gray')
plt.title('Sobel Filter')
plt.axis('off')
Sobel filter. Image provided by the author.
cv.Sobel（src， ddepth， dx， dy[， dst[， ksize[， scale[， delta[， borderType]]]]]） -> dst
dxand: the order of the derivatives in the x and y directions respectively. These values determine the direction in which the gradient is calculated. For example, set and calculate the gradient in the x direction.dydx=1dy=0ksize(optional): The size of the Sobel kernel used for calculations. It is usually set to an odd number, such as,,, etc. If not provided, the default value is, which corresponds to a 3x3 core.1353scale(Optional): Scale factor to apply to calculated Sobel values. This parameter helps increase or decrease the effect of the Sobel operator.delta(Optional): Value to add to the calculated Sobel value. It can be used to control the overall brightness or contrast of the output.
5.2 Prewett filter
Similar to the Sobel filter, the Prewitt filter also calculates the gradient of the image intensity to detect edges. It uses a simpler kernel than the Sobel kernel but is still effective at highlighting edges.
- The strength of the filter should be carefully adjusted to avoid over-sharpening the image.
- If possible, the filter should be applied to a noise-free version of the image.
- Filters should be used in conjunction with other image processing techniques (such as smoothing) to improve the overall quality of the image.
Prewett operator. source
image = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
# Prewitt Filter
kernel_x = np.array([[-1, 0, 1], [-1, 0, 1], [-1, 0, 1]])
kernel_y = np.array([[-1, -1, -1], [0, 0, 0], [1, 1, 1]])
prewitt_x = cv2.filter2D(image, cv2.CV_64F, kernel_x)
prewitt_y = cv2.filter2D(image, cv2.CV_64F, kernel_y)
prewitt_edges = np.sqrt(prewitt_x**2 + prewitt_y**2)
# plot
plt.figure(figsize=(12, 8))
plt.subplot(2, 3, 1)
plt.imshow(image, cmap='gray')
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 3, 2)
plt.imshow(prewitt_edges, cmap='gray')
plt.title('Prewitt Filter')
plt.axis('off')
Prewett filter. Image provided by the author.
5.3 Laplacian of Gaussian filtering
The LoG filter combines the effects of blurring and edge detection. It first applies Gaussian blur to the image and then calculates the Laplacian operator on the blurred image. The result is that the edges of the image are enhanced and noise is suppressed.
Laplacian of Gaussian filter. source
The LoG filter is a separable filter, which means it can be calculated by convolving the image with a single kernel (i.e. a 3x3 matrix). The kernel consists of weights designed to enhance the edges of the image.
- The strength of the filter should be carefully adjusted to avoid over-sharpening the image.
- If possible, the filter should be applied to a noise-free version of the image.
- Filters should be used in conjunction with other image processing techniques (such as smoothing) to improve the overall quality of the image.
image = cv2.imread(path, cv2.IMREAD_GRAYSCALE)
# Apply Laplacian of Gaussian (LoG) Filter
log_edges = cv2.Laplacian(image, cv2.CV_64F)
log_edges = cv2.convertScaleAbs(log_edges)
# plot
plt.figure(figsize=(12, 8))
plt.subplot(2, 3, 1)
plt.imshow(image, cmap='gray')
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 3, 2)
plt.imshow(log_edges, cmap='gray')
plt.title('LoG Filter')
plt.axis('off')
LoG filter. Image provided by the author.
The Laplacian operation can generate both positive and negative values, which may not be suitable for visualization. Function is used to convert the output of the Laplacian operation to an absolute value and then to a data type more suitable for image display ().cv2.convertScaleAbs(log_edges)uint8
5.4 Smart edge detector
Canny edge detector is a popular image processing technique used to detect edges in images.
Smart edge detector. source
Canny edge detectors work by identifying areas in an image with rapidly changing intensity, which often correspond to object boundaries or important features.
The output of the Canny edge detector is a binary image where edge pixels are marked with white and non-edge pixels with black. The technique is versatile and can be fine-tuned by adjusting parameters such as the standard deviation of the Gaussian blur and the edge detection threshold.
canny_edges = cv2.Canny(image, 30, 60)
# plot
plt.figure(figsize=(12, 8))
plt.subplot(2, 3, 1)
plt.imshow(image, cmap='gray')
plt.title('Original Image')
plt.axis('off')
plt.subplot(2, 3, 2)
plt.imshow(canny_edges, cmap='gray')
plt.title('Canny Edge Filter')
plt.axis('off')
Smart edge filter. Image provided by the author.
cv.Canny(Image, Threshold1, Threshold2[, Edges[, ApertureSize[, L2Gradient]]]) -> Edges
cv.Canny（dx， dy， threshold1， threshold2[， edges[， L2gradient]]） -> edge
image: The input image to which Canny edge detection will be applied. It is usually a grayscale image.threshold1and: These are the lower and upper thresholds used in the dual threshold step. The Canny algorithm marks pixels with gradient magnitudes above as strong edges and pixels between edges as weak edges. Pixels below are considered non-edge and will be discarded.threshold2threshold2threshold1threshold2threshold1apertureSize(Optional): The size of the Sobel kernel used for gradient calculations. It is usually set to , or . If not provided, the default value is.3573L2gradient(optional): Flag indicating whether the L2 norm should be used to calculate gradient magnitude. If , then use Euclidean distance. If , the L1 norm is used. The default value is .TrueFalseFalse
At its core, image filtering is about extracting hidden insights from pixel data. Through the mathematical power of convolution, it enhances edges, reduces noise, and transforms raw images into engaging stories. This process combines purpose with technology, solidifying image filtering's powerful ability to reveal the unseen and enhance visual perception. Okan Jernigan