According to the different levels of image representation, image fusion can be divided into three levels of fusion: pixel-level fusion, feature-level fusion and decision-level fusion. The following figure is the image fusion level division diagram.
Among them, pixel-level fusion is located at the lowest level, which can be regarded as only extracting features from the information and using it directly. It is precisely because of its maximum retention of information that it is superior to the other two levels in terms of accuracy and robustness. In contrast, pixel-level fusion obtains richer detailed information and is the most commonly used fusion method. Therefore, it is also a hot research topic in the field of image fusion. At the same time, due to the high registration accuracy it requires, it must reach the pixel level. Therefore, pixel-level image fusion technology has high requirements on equipment, and the fusion process is time-consuming and difficult to process in real time. Pixel-level fusion is generally completed in four steps: preprocessing, transformation, synthesis and inverse transformation.
The task I currently need to deal with is the fusion and segmentation of Sar and Optical optical images. I used two methods for fusion: the ISH color space fusion method based on non-multiscale transformation , and the image fusion method based on Laplace pyramid transform.
ISH color space fusion method based on non-multiscale transformation
The IHS color model is suitable for people's intuitive color matching methods, and thus has become the most commonly used color model for color image processing. Intensity represents the overall brightness of the spectrum, corresponding to the spatial resolution of the image. Hue describes the properties of pure color, which is determined by the main wavelength of the spectrum. It is the qualitative difference between the spectrum. Saturation represents the proportion of the main wavelength of the spectrum in intensity. , hue and saturation represent the spectral resolution of the image.
Compared with the RGB model, the IHS model is more consistent with the way the human eye describes and interprets colors. At the same time, because the three basic features of I, H, and S are independent of each other, the IHS model is often used for image processing based on color description. In this method, the color information (hue and saturation) and colorless light intensity information (brightness) carried in the color image are processed separately.
IHS transformation formula
The IHS spatial image fusion method takes advantage of the IHS model in representing color images and is often used for fusion processing of remote sensing images. In order to preserve the spectral information of a multispectral image while enhancing its spatial resolution, low spatial resolution multispectral images and high spatial resolution panchromatic images can be fused in the IHS space to convert the multispectral image from the RGB image model to the IHS model. , and fuse the component reflecting the spatial resolution I of the multispectral image with the panchromatic image in the IHS space, and then inversely transform the fusion result back into the RGB space to obtain a multispectral image with improved spatial resolution after fusion.
ISH image fusion processing steps:
① Convert the R, G, and B bands of the multispectral image to the IHS space to obtain the three components of I, H, and S;
② The brightness component I obtained after IHS transformation of the panchromatic image and the multispectral image, at a certain Fusion is performed under the fusion rules to obtain a new brightness component (fusion component) I';
③ Replace the brightness component with the fusion component I' obtained in step 2, and convert it to the RGB space together with the H and S component images, and finally obtain the fusion image.
ISH image fusion program:
img_process.py
import cv2
from PIL import Image
import numpy as np
from osgeo import gdal
class ImgProcess:
def IHS(self, data_low, data_high):
"""
基于IHS变换融合算法
输入：np.ndArray格式的三维数组
返回：可绘出图像的utf-8格式的三维数组
"""
A = [[1. / 3., 1. / 3., 1. / 3.], [-np.sqrt(2) / 6., -np.sqrt(2) / 6., 2 * np.sqrt(2) / 6],
[1. / np.sqrt(2), -1. / np.sqrt(2), 0.]]
# RGB－>IHS正变换矩阵
B = [[1., -1. / np.sqrt(2), 1. / np.sqrt(2)], [1., -1. / np.sqrt(2), -1. / np.sqrt(2)], [1., np.sqrt(2), 0.]]
# IHS－>RGB逆变换矩阵
A = np.matrix(A)
B = np.matrix(B)
band, w, h = data_high.shape
pixels = w * h
data_low = data_low.reshape(3, pixels)
data_high = data_high.reshape(3, pixels)
a1 = np.dot(A, np.matrix(data_high)) # 高分影像正变换
a2 = np.dot(A, np.matrix(data_low)) # 低分影像正变换
a2[0, :] = a2[0, :] *0.3 + a1[0, :]*0.7 # 用高分影像第一波段替换低分影像第一波段
# a2[0, :] = a1[0, :] # 用高分影像第一波段替换低分影像第一波段
RGB = np.array(np.dot(B, a2)) # 融合影像逆变换
RGB = RGB.reshape((3, w, h))
min_val = np.min(RGB.ravel())
max_val = np.max(RGB.ravel())
RGB = np.uint8((RGB.astype(np.float) - min_val) / (max_val - min_val) * 255)
RGB = Image.fromarray(cv2.merge([RGB[0], RGB[1], RGB[2]]))
return RGB
def imresize(self, data_low, data_high):
"""
图像缩放函数
输入：np.ndArray格式的三维数组
返回：np.ndArray格式的三维数组
"""
band, col, row = data_high.shape
data = np.zeros(((band, col, row)))
for i in range(0, band):
# data[i] = smi.imresize(data_low[i], (col, row))
data[i] = cv2.resize(data_low[i], (col, row))
return data
def gdal_open(self, path):
"""
读取图像函数
输入：图像路径
返回：np.ndArray格式的三维数组
"""
data = gdal.Open(path)
col = data.RasterXSize # 读取图像长度
row = data.RasterYSize # 读取图像宽度
data_array_r = data.GetRasterBand(1).ReadAsArray(0, 0, col, row).astype(np.float) # 读取图像第一波段并转换为数组
data_array_g = data.GetRasterBand(2).ReadAsArray(0, 0, col, row).astype(np.float) # 读取图像第二波段并转换为数组
data_array_b = data.GetRasterBand(3).ReadAsArray(0, 0, col, row).astype(np.float) # 读取图像第三波段并转换为数组
data_array = np.array((data_array_r, data_array_g, data_array_b))
return data_array
main.py
from tools.img_process import ImgProcess
if __name__ == "__main__":
img_p = ImgProcess()
path_low = '/home/workstation8/Research/segformer-pytorch-diploma/VOCdevkit/DFC2023/JPEGImages'
path_high = '/home/workstation8/Research/segformer-pytorch-diploma/VOCdevkit/DFC2023/SAR-JPEGImages-3c-temp'
save_path = '/home/workstation8/Research/segformer-pytorch-diploma/VOCdevkit/DFC2023/IHSFusion-O-S-I/'
img_p.makeDir(save_path)
for img_name in os.listdir(path_high):
data_low = img_p.gdal_open(os.path.join(path_low, img_name))
data_high = img_p.gdal_open(os.path.join(path_high, img_name))
data_low = img_p.imresize(data_low, data_high)
RGB = img_p.IHS(data_low, data_high)
RGB.save(os.path.join(save_path, img_name))
Image fusion method based on Laplace pyramid transform
Introduction to Image Pyramid
Image pyramid is a type of multi-scale expression, an effective but conceptually simple structure for interpreting images at multiple resolutions. A pyramid of images is a collection of gradually decreasing resolutions arranged in a pyramid shape and originating from the same original image. Obtained through ladder down sampling, sampling does not stop until a certain termination condition.
To put it bluntly, the image pyramid is image scaling dressed up as a pyramid. Generally, there are Gaussian image pyramid and Laplacian image pyramid.
Laplacian Pyramid (Bandpass Pyramid)
Laplacian Pyramid: Used to reconstruct images, that is, predict residuals, to restore the image to the greatest extent.
The principle of reconstructing a small image into a large image: subtract the predicted image after upsampling and Gaussian convolution of the previous layer image from each layer of the Gaussian pyramid, and obtain a series of difference images, which are LP decomposed images. .
The Laplacian pyramid actually saves the missing information by calculating the result of downsampling and then upsampling the image and the residual of the original image. The formula is:
L ( i ) = G ( i ) − P y r U p ( G ( i + 1 ) ) L ( i ) = G ( i ) − P y r U p ( G ( i + 1 ) ) L ( i ) = G ( i ) − P y r U p ( G ( i + 1 ) ) L ( i ) = G ( i ) − P y r U p ( G ( i + 1 ) ) L(i) = G(i) - PyrUp(G(i + 1)) L(i)=G(i)−PyrUp(G(i+1)) L(i)=G(i)−PyrUp(G(i+1))L(i)=G(i)−PyrUp(G(i+1))L(i)=G(i)−PyrUp(G(i+1))
That is to say, the Laplacian pyramid is actually a pyramid composed of the above residual image, which prepares for the restored image. After obtaining the Laplacian pyramid of each image, the images at the corresponding levels need to be fused to finally restore the image. Laplacian pyramid can accurately restore picture information. The process of restoring the image is the process of reconstruction.
Laplace pyramid fusion steps:
① Perform Gaussian downsampling on the two pictures respectively
② Find the Laplace pyramid of the two pictures respectively
③ Fusion of the Laplace pyramid of the two pictures according to a certain weight
④ Reconstruct the fused Laplace pyramid to obtain the fused image
L ’ = W 1 × L s + W 2 × L o L’= W~1~ × L~s~ + W~2~ × L~o~ L’=W 1 ×L s +W 2 ×L o
In this fusion project, the weight fusion in step ③ above is: after calculating the Laplace pyramid of the SAR and OPT images respectively, then adding and fusion with a certain weight w1 and w2. After many parameter adjustments, it was found that the best features were extracted when w1:w2 = 0.2:0.8.
Laplace Pyramid Fusion Procedure
fusion_process.py
# -*- coding:utf-8 -*-
__author__ = 'Microcosm'
import cv2
import matplotlib as mpl
import os
mpl.use('TKAgg')
class LP_Fusion():
def makeDir(self, path):
"""
如果path路径不存在，则创建
:param path:
:return:
"""
if not os.path.exists(path):
os.makedirs(path)
def sameSize(self, img1, img2):
"""
使得img1的大小与img2相同
"""
rows, cols, dpt = img2.shape
dst = img1[:rows, :cols]
return dst
def pyrDown(self, img, layer):
"""
下采样生成高斯金字塔
Args:
img:
layer:
Returns:
"""
G = img.copy()
gp_img = [G]
for i in range(layer):
G = cv2.pyrDown(G)
gp_img.append(G)
return gp_img
def get_lp(self, gp_img, layer):
"""
生成拉普拉斯金字塔【等于「高斯金字塔中的第i层」与「高斯金字塔中的第i+1层的向上采样结果」之差】
Args:
gp_img:
layer:
Returns:
"""
lp_img = [gp_img[layer]]
for i in range(layer, 0, -1): # [6, 5, 4, 3, 2, 1]
GE = cv2.pyrUp(gp_img[i])
L = cv2.subtract(gp_img[i - 1], self.sameSize(GE, gp_img[i - 1]))
lp_img.append(L)
return lp_img
def fusion(self, lp_img_1, lp_img_2):
"""
图像融合重建
Args:
lp_img_1:
lp_img_2:
Returns:
"""
LS = []
for la, lb in zip(lp_img_1, lp_img_2):
rows, cols, dpt = la.shape
ls = la
# print(type(ls))
ls = la*0.8 + lb*0.2
LS.append(ls)
return LS
def pyrUp(self, LS, layer):
"""
上采样 并与 拉普拉斯相加 恢复原始状态
Args:
LS:
layer:
Returns:
"""
ls_reconstruct = LS[0]
for i in range(1, layer):
# print(LS[i])
ls_reconstruct = cv2.pyrUp(ls_reconstruct)
ls_reconstruct = cv2.add(self.sameSize(ls_reconstruct, LS[i]), LS[i])
# print("ls_reconstruct {}".format(ls_reconstruct.shape))
return ls_reconstruct
main.py
from tools.fusion_process import LP_Fusion
import cv2
import os
from tqdm import tqdm, trange
from matplotlib import pyplot as plt
if __name__ == "__main__":
lp_p = LP_Fusion()
# 层数-1
layer = 1
# img_p = ImgProcess()
img1_path = '/home/workstation8/Research/segformer-pytorch-diploma/VOCdevkit/DFC2023/JPEGImages'
img2_path = '/home/workstation8/Research/segformer-pytorch-diploma/VOCdevkit/DFC2023/SAR-JPEGImages'
save_path = '/home/workstation8/Research/segformer-pytorch-diploma/VOCdevkit/DFC2023/LPFusion/LPFusion-{}L-8O-2S'.format(layer)
layer -= 1
lp_p.makeDir(save_path)
for index in tqdm(range(len(os.listdir(img2_path))), colour='BLUE'):
img_name_list = os.listdir(img2_path)
img_1 = cv2.imread(os.path.join(img1_path, img_name_list[index]))
img_2 = cv2.imread(os.path.join(img2_path, img_name_list[index]))
# 对img_1进行layer层高斯降采样
gp_img_1 = lp_p.pyrDown(img_1, layer=layer)
# 对img_2进行layer层高斯降采样
gp_img_2 = lp_p.pyrDown(img_2, layer=layer)
# 求img_1的Laplace金字塔
lp_img_1 = lp_p.get_lp(gp_img_1, layer)
# 求img_2的Laplace金字塔
lp_img_2 = lp_p.get_lp(gp_img_2, layer)
# # 对img_1和img_2的Laplace金字塔进行融合
LS = lp_p.fusion(lp_img_1, lp_img_2)
# 对融合后的Laplace金字塔重建获取融合后的结果
ls_reconstruct = lp_p.pyrUp(LS, layer+1)
# 保存
cv2.imwrite(os.path.join(save_path, img_name_list[index]), ls_reconstruct)
# # 可视化
# new_img = plt.imread(os.path.join(save_path, img_name))
# plt.imshow(new_img)
# plt.show()
Reference blog:
Summary of image fusion methods
Common methods for pixel-level image fusion
Image pyramid to achieve image fusion