Table of contents
(3) Processing of Gaussian blur
(4) Processing of motion blur + Gaussian noise
1. Background description
Image deblurring is a classic problem in low-level computer vision. Its purpose is to restore a clear image from a blurred input image. It aims to reduce or eliminate factors such as camera shake, object motion or lens distortion through algorithms and mathematical models. causing image blur. This technique is often used to improve the clarity and visualization quality of images, making them easier to analyze, identify, or use in other applications. Image deblurring has important application value in fields such as photography, medical imaging, astronomy, security monitoring, and computer vision.
What is image blur? Generally speaking, image blur is caused by various factors in the process of capturing images, including camera shake, target movement, and defocus. According to the difference of blurred pictures, blurred pictures are generally divided into the following categories: motion blur, defocus blur, Gaussian blur, and mixed blur.
(1) Motion Blur: Under sufficient lighting conditions, when the exposure time is short enough, the camera can capture a clear image. However, when the exposure time is too long relative to the movement of the object or camera, the image will be blurred, and the blurred picture is generally called motion blur.
(2) Defocus Blur: In addition to motion blur, image clarity is also affected by the target position and the focal length of the camera. In the camera's imaging area, the depth of field of different targets is different. When the camera's focus system cannot focus on certain targets, the camera will take out-of-focus and blurred pictures.
(3) Gaussian Blur: A common blur method that reduces the high-frequency details of the image by performing Gaussian filtering and Gaussian convolution on the image, thereby making the image look smoother.
(4) Mix Blur: When a picture is affected by multiple factors at the same time, the resulting blur is a mixed blur. For example, when a camera shoots a high-speed moving object in an out-of-focus state, the blur obtained is a mixed blur. .
Here are several specific examples of ambiguity:
In order to evaluate the effect of deblurring, some image evaluation algorithms are usually used, which include subjective-based and objective-based evaluations. The subjective method does not require a clear image for reference. A representative metric is MOS, where reviewers rate image quality on a scale of 1-5 and then average the scores. For the problem of image deblurring, most existing methods are evaluated based on objective-based metrics, which can be further divided into two categories: referenced and non-referenced metrics. Reference indicators evaluate image quality by comparing the restored image with the clear image, including PSNR, SSIM, WSNR, MS-SSIM, IFC, NQM, UIQI, VIT, LPIPS, etc., among which PSNR and SSIM are the most commonly used index. Unlike referenced metrics, reference-free metrics measure the quality using only deblurred images. Common indicators include BIQI, BLINDS, BRISQUE, CORNIA, NIQE, SSEQ, etc. In addition, the performance of image deblurring algorithms has also been evaluated by measuring the impact of the deblurring algorithm on the accuracy of different vision tasks such as object detection and recognition. Since the main purpose of our major assignment is to write a tutorial blog on how to implement the image deblurring method for future students who will choose the digital image processing course to learn, there is no need to limit the evaluation indicators, so our indicators ( metric ) , several of our team members subjectively judge the effect of image deblurring.
Since when we use electronic products such as mobile phones and cameras to shoot in our daily lives, the pictures we get are very prone to image blurring. Therefore, looking for algorithms or methods that can achieve image deblurring is very important for our real-world applications. It is of great significance.
2. Technology application
In this experiment, we mainly use Wiener filtering in deconvolution to achieve image deblurring.
The field of deconvolution is large and complex. For motion blur/PSF blur/noise/different imaging fields, the usage is different. This article summarizes deconvolution in the field of incoherent imaging with point spread function PSF ( Point Spread Function) blur. method. Let’s introduce what the deconvolution method is.
An ideal lens does not exist in the real world. For example, for pinhole imaging, due to the wave nature of light, if the pinhole is too small, light diffraction will occur, resulting in blurred imaging, and a very low signal-to-noise ratio. The image is noisy. On the contrary, if the hole is too large, the light from the same image point will come from multiple object points, which will also cause the image to be blurred. Therefore, only a certain balanced size between the two can make the image both clearer and with lower noise. There are still many deficiencies in imaging in the real world, so there are always some flaws in the lens that will cause one object point to be projected into many points. The image formed by an ideal point passing through the camera is described by the point spread function PSF (Point Spread Function). When we consider that the lens only has the defect of diffraction, the PSF of an ideal small point after passing through the circular aperture will show a special shape. This is an image called Airy Pattern. This kind of PSF formed by diffraction We call it diffraction limit PSF (Diffraction Limit PSF). And if you perform Fourier transform on this PSF, you can get the optical transfer function OTF (Optical Transfer Function) of this lens. PSF or OTF have an impact on imaging. We take PSF as the blur kernel C as an example:
When we have the imaging b of the actual lens, and also know the PSF of the lens (i.e. C in the picture), if we can find a way to get X with known b and c, we can achieve image deblurring, and this The method is the deconvolution method.
For deconvolution methods, it can be divided into blind deconvolution (not knowing the blur kernel) and non-blind deconvolution (knowing the blur kernel). This major assignment mainly uses the Wiener deconvolution method, one of the most representative and widely used non-blind deconvolution methods.
In the real world, the existence of noise needs to be considered, then we let n be noise, then the formula can be rewritten as:
Convolution in the spatial domain corresponds to multiplication in the frequency domain, so:
Now we can think of the deconvolution problem as finding a frequency domain function H(ω) that minimizes the following expected error:
So the question is transformed into:
Bring in the expression for B:
Expanding this formula we have:
Here, because the noise N is not related to the image X, then
Because the mean value of noise is 0, so E[XN] = 0
In this way we can write the above minimization expression as:
Find the gradient for the following loss :
And assuming ϑLoss / ϑH =0 , then we have
Divide the numerator and denominator at the same time 
to get:
Then the Wiener filter is
Rewrite C as F(c) and B as F(b), so we get the Wiener filter expression we introduced above:
Wiener filter can also be rewritten as :
G(f)) and H(f) are the Fourier transforms of g and h in the frequency domain ff
S(f) is the power spectrum of the input signal x(t)
N(f) is the power spectrum of n(t) of the noise
* stands for complex conjugate.
The filtering process can be completed in the frequency domain: X^(f)=G(f)*Y(f), where X^(f) is the Fourier transform of x^(t), which can be obtained by inverse Fourier transform The result after deconvolution x^(t).
After simplification, we can get:
Above we regarded the problem as a minimization problem in the frequency domain, but we can also regard the problem as a minimization problem in the spatial domain (L2 regularization):
We found that after regularization, the noise of the image was greatly reduced and the image quality was better.
Not only that, we can also use other regularizations such as L1 regularization.
3. Actual processing
(1) Processing of motion blur
import numpy as np
from numpy.fft import fft2, ifft2
import cv2
#计算得到卷积运动模糊核PSF
def motion_blur(image,image_size, angle, degree):
image = np.array(image)
# 这里生成任意角度的运动模糊kernel的矩阵， degree越大，模糊程度越高
M = cv2.getRotationMatrix2D((degree / 2, degree / 2), angle, 1)
motion_blur_kernel = np.diag(np.ones(degree))
motion_blur_kernel = cv2.warpAffine(motion_blur_kernel, M, (degree, degree))
motion_blur_kernel = motion_blur_kernel / degree
return motion_blur_kernel / motion_blur_kernel.sum()
'''
blurred = cv2.filter2D(image, -1, motion_blur_kernel)
# convert to uint8
cv2.normalize(blurred, blurred, 0, 255, cv2.NORM_MINMAX)
blurred = np.array(blurred, dtype=np.uint8)
return blurred
'''
#给清晰图片加上卷积模糊核，给图片制造模糊
def apply_blur(input_img, psf, epsilon):
blurred = cv2.filter2D(input_img, -1, psf)
# convert to uint8
cv2.normalize(blurred, blurred, 0, 255, cv2.NORM_MINMAX)
blurred = np.array(blurred, dtype=np.uint8)
return blurred
#维纳逆波实现图像去除模糊
def apply_wiener(input_img, psf, epsilon, k=0.001):
#psf /= np.sum(psf)
dummy = np.copy(input_img)
dummy = fft2(dummy)
psf = fft2(psf, s = input_img.shape)
psf = np.conj(psf) / (np.abs(psf) ** 2 + k)
dummy = dummy * psf
dummy = np.abs(ifft2(dummy))
return dummy
#数组裁切
def clip_and_cast(array):
array = np.where(array < 0, 0, array)
array = np.where(array > 255, 255, array)
array = array.astype(np.uint8)
return array
#对运动模糊进行单一通道进行处理
def motion_main_process(input_image):
motion_blur_channels = []
img_height, img_width = input_image.shape[:2]
# 运动模糊模糊核
motion_blur_psf = motion_blur(input_image,(img_height, img_width), 45, 35)
motion_blurred_result = np.abs(apply_blur(input_image, motion_blur_psf, 1e-3))
motion_wiener_result = apply_wiener(motion_blurred_result, motion_blur_psf, 1e-3)
motion_blur_channels.append((clip_and_cast(motion_blurred_result), clip_and_cast(motion_wiener_result)))
return motion_blur_channels
if __name__ == '__main__':
input_image = cv2.imread('lena1960.jpg')
b_channel, g_channel, r_channel = cv2.split(input_image.copy())
motion_final_result = []
for channel in [b_channel, g_channel, r_channel]:
processed_channel = motion_main_process(channel)
motion_final_result.append(processed_channel)
motion_blurred_img = cv2.merge([motion_final_result[0][0][0], motion_final_result[1][0][0], motion_final_result[2][0][0]])
wiener_deblurred_img = cv2.merge([motion_final_result[0][0][1], motion_final_result[1][0][1], motion_final_result[2][0][1]])
cv2.imwrite('Motion_Blurred_Image.JPG', motion_blurred_img)
cv2.imwrite('Wiener_Motion_Image.JPG', wiener_deblurred_img)
Processing effect
(2) Processing of mean fuzzy
import numpy as np
from numpy.fft import fft2, ifft2
import cv2
#计算得到卷积均值模糊核PSF
def mean_blur(kernel_size):
if kernel_size % 2 == 0:
kernel_size += 1 # Ensure kernel size is odd
kernel = np.ones((kernel_size, kernel_size), dtype=np.float32) / kernel_size**2
PSF =kernel/np.sum(kernel)
return PSF
#给清晰图片加上卷积模糊核，给图片制造模糊
def apply_blur(input_img, psf, epsilon):
blurred = cv2.filter2D(input_img, -1, psf)
# convert to uint8
cv2.normalize(blurred, blurred, 0, 255, cv2.NORM_MINMAX)
blurred = np.array(blurred, dtype=np.uint8)
return blurred
#维纳逆波实现图像去除模糊
def apply_wiener(input_img, psf, epsilon, k=1e-3):
#psf /= np.sum(psf)
dummy = np.copy(input_img)
dummy = fft2(dummy)
psf = fft2(psf, s = input_img.shape)
psf = np.conj(psf) / (np.abs(psf) ** 2 + k)
dummy = dummy * psf
dummy = np.abs(fft.fftshift(ifft2(dummy)))
return dummy
#数组裁切
def clip_and_cast(array):
array = np.where(array < 0, 0, array)
array = np.where(array > 255, 255, array)
array = array.astype(np.uint8)
return array
#对均值模糊进行单一通道进行处理
def mean_main_process(input_image):
mean_blur_channels = []
# 均值模糊模糊核
mean_blur_psf = mean_blur(25)
mean_blurred_result = np.abs(apply_blur(input_image, mean_blur_psf, 1e-3))
mean_wiener_result = apply_wiener(mean_blurred_result, mean_blur_psf, 1e-3)
mean_blur_channels.append((clip_and_cast(mean_blurred_result), clip_and_cast(mean_wiener_result)))
return mean_blur_channels
if __name__ == '__main__':
input_image = cv2.imread('lena1960.jpg')
b_channel, g_channel, r_channel = cv2.split(input_image.copy())
mean_final_result = []
for channel in [b_channel, g_channel, r_channel]:
processed_channel = mean_main_process(channel)
mean_final_result.append(processed_channel)
mean_blurred_img = cv2.merge([mean_final_result[0][0][0], mean_final_result[1][0][0], mean_final_result[2][0][0]])
wiener_deblurred_img = cv2.merge([mean_final_result[0][0][1], mean_final_result[1][0][1], mean_final_result[2][0][1]])
cv2.imwrite('Motion_Blurred_Image.JPG', mean_blurred_img)
cv2.imwrite('Wiener_Motion_Image.JPG', wiener_deblurred_img)
Processing effect
(3) Processing of Gaussian blur
import numpy as np
from numpy.fft import fft2, ifft2
from scipy.signal import gaussian
import cv2
#得到卷积高斯模糊核PSF
def gaussian_kernel(kernel_size ):
h = gaussian(kernel_size, kernel_size / 3).reshape(kernel_size, 1)
h = np.dot(h, h.transpose())
h /= np.sum(h)
return h
#给清晰图片加上卷积模糊核，给图片制造高斯模糊
def apply_blur(input_img, psf, epsilon):
blurred = cv2.filter2D(input_img, -1, psf)
# convert to uint8
cv2.normalize(blurred, blurred, 0, 255, cv2.NORM_MINMAX)
blurred = np.array(blurred, dtype=np.uint8)
return blurred
#维纳逆波实现图像去除模糊
'''
def apply_wiener(input_img, psf, epsilon, k=0.001):
s = psf.shape
input_fft = fft.fft2(input_img, s)
psf_fft = fft.fft2(psf) + epsilon
psf_fft_1 = np.conj(psf_fft) / (np.abs(psf_fft) ** 2 + k)
result = fft.ifft2(input_fft * psf_fft_1)
result = np.abs(fft.fftshift(result))
return result
'''
def apply_wiener(input_img, psf, epsilon, k=0.001):
#psf /= np.sum(psf)
dummy = np.copy(input_img)
dummy = fft2(dummy)
psf = fft2(psf, s = input_img.shape)
psf = np.conj(psf) / (np.abs(psf) ** 2 + k)
dummy = dummy * psf
dummy = np.abs(ifft2(dummy))
return dummy
#数组裁切
def clip_and_cast(array):
array = np.where(array < 0, 0, array)
array = np.where(array > 255, 255, array)
array = array.astype(np.uint8)
return array
#对高斯模糊进行单一通道进行处理
def gaussian_main_process(input_image):
gaussian_blur_channels = []
img_height, img_width = input_image.shape[:2]
# 高斯模糊模糊核
gaussian_blur_psf = gaussian_kernel(15)
gaussian_blurred_result = np.abs(apply_blur(input_image, gaussian_blur_psf, 1e-3))
new_gaussian_blur_psf = gaussian_kernel(15)
gaussian_wiener_result = apply_wiener(gaussian_blurred_result, new_gaussian_blur_psf, 1e-3)
gaussian_blur_channels.append((clip_and_cast(gaussian_blurred_result), clip_and_cast(gaussian_wiener_result)))
return gaussian_blur_channels
if __name__ == '__main__':
input_image = cv2.imread('lena1960.jpg')
b_channel, g_channel, r_channel = cv2.split(input_image.copy())
gaussian_final_result = []
for channel in [b_channel, g_channel, r_channel]:
processed_channel = gaussian_main_process(channel)
gaussian_final_result.append(processed_channel)
gaussian_blurred_img = cv2.merge([gaussian_final_result[0][0][0], gaussian_final_result[1][0][0], gaussian_final_result[2][0][0]])
wiener_deblurred_img = cv2.merge([gaussian_final_result[0][0][1], gaussian_final_result[1][0][1], gaussian_final_result[2][0][1]])
cv2.imwrite('Gaussian_Blurred_Image.JPG', gaussian_blurred_img)
cv2.imwrite('Wiener_Deblurred_Image.JPG', wiener_deblurred_img)
Processing effect
(4) Processing of motion blur + Gaussian noise
import os
import numpy as np
from numpy.fft import fft2, ifft2
from scipy.signal import gaussian, convolve2d
import matplotlib.pyplot as plt
import cv2
#添加运动模糊
def add_motion_blur(img, kernel_size = 3):
dummy = np.copy(img)
h = np.eye(kernel_size) / kernel_size
dummy = convolve2d(dummy, h, mode = 'valid')
return dummy
#添加高斯噪声
def add_gaussian_niose(img, sigma):
gauss = np.random.normal(0, sigma, np.shape(img))
noisy_img = img + gauss
noisy_img[noisy_img < 0] = 0
noisy_img[noisy_img > 255] = 255
return noisy_img
#维纳滤波器
def wiener_filter(img, kernel, K):
kernel /= np.sum(kernel)
dummy = np.copy(img)
dummy = fft2(dummy)
kernel = fft2(kernel, s = img.shape)
kernel = np.conj(kernel) / (np.abs(kernel) ** 2 + K)
dummy = dummy * kernel
dummy = np.abs(ifft2(dummy))
return dummy
#高斯核
def gaussian_kernel(kernel_size = 3):
h = gaussian(kernel_size, kernel_size / 3).reshape(kernel_size, 1)
h = np.dot(h, h.transpose())
h /= np.sum(h)
return h
#如果想将输入的彩色图片转换成灰色图片后进行模糊化和去模糊化，可以在读取图片时使用这个函数
def rgb2gray(rgb):
return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140])
if __name__ == '__main__':
input_image = cv2.imread('lena1960.jpg')
# 将图像划分成三通道
b_channel, g_channel, r_channel = cv2.split(input_image.copy())
# 在这个循环内给每个通道用 blur(img, kernel_size=15) 添加运动模糊
motion_blurred_channels = [add_motion_blur(channel, kernel_size=15) for channel in [b_channel, g_channel, r_channel]]
motion_blurred_img = cv2.merge(motion_blurred_channels)
# 将经过运动模糊后图像划分成三通道
mb_channel, mg_channel, mr_channel = cv2.split(motion_blurred_img.copy())
# 在这个循环内给每个通道用 add_gaussian_noise(img, sigma=20) 添加高斯噪声
motion_gaussian_blurred_channels = [add_gaussian_niose(channel, sigma=20) for channel in [mb_channel, mg_channel, mr_channel]]
motion_gaussian_blurred_img = cv2.merge(motion_gaussian_blurred_channels)
# 保存运动模糊和加入高斯噪声后的运动模糊图像
cv2.imwrite('Motion_Blurred_Image.JPG', motion_blurred_img)
cv2.imwrite('Gaussian_Blurred_Image.JPG', motion_gaussian_blurred_img)
# 使用Wiener滤波器进行运动模糊图像的去模糊
estimated_kernel = gaussian_kernel(15) # 估计的模糊核
motion_filtered_channels = [wiener_filter(channel, estimated_kernel, K=0.01) for channel in motion_blurred_channels]
motion_filtered_img = cv2.merge(motion_filtered_channels)
cv2.imwrite('Wiener_Motion_Deblurred_Image.JPG', motion_filtered_img)
# 使用Wiener滤波器进行高斯模糊图像的去模糊
new_estimated_kernel = gaussian_kernel(3) # 估计的模糊核
motion_gaussian_filtered_channels = [wiener_filter(channel, new_estimated_kernel, K=0.01) for channel in motion_gaussian_blurred_channels]
motion_gaussian_filtered_img = cv2.merge(motion_gaussian_filtered_channels)
cv2.imwrite('Wiener_Gaussian_Motion_Deblurred_Image.JPG', motion_gaussian_filtered_img)
Processing effect
4. Ringing phenomenon
In image processing, an image is filtered. If the frequency domain filter selected has a steep change, the filtered image will cause "ringing".
The possible reasons are as follows:
1) Filter noise ringing En: There are inevitable measurement and quantization errors in the observed pixels.
2) Filter deviation ringing Ed: The point spread function PSF frequency domain response has zero points. The regularization process converts an ill-posed problem into a well-posed problem, resulting in the deviation of the deconvolution of the restored filter and the original PSF.
3) Boundary truncation ringing Eb: The gradient of the image changes drastically at the boundary. When Fourier transform is performed on the image, periodic continuation causes the gradient at the boundary to change drastically, causing ringing.
After searching for information on the Internet, I found that the extended boundary preprocessing method (denoising and smoothing) can be used to effectively eliminate boundary ringing.
5. Summary
The bandpass frequency of Wiener filter depends on the signal-to-noise ratio SNR(f)=S(f)/N(f). When the signal-to-noise ratio approaches infinity (that is, the noise is zero), G(f) increases, and Wiener filtering is simplified into inverse filtering; when the noise increases, the signal-to-noise ratio decreases, and G(f) also decreases. Therefore, we can appropriately adjust the value of the signal-to-noise ratio, which will help improve the restoration effect of noise-blurred images.