Abstract:
Infrared and visible light image fusion is an important branch of heterogeneous sensor information fusion, which has important research significance in military, remote sensing and other fields. Starting from the characteristics of infrared and visible light images, this article introduces the current status of the two image fusions, summarizes commonly used image fusion methods based on multi-scale analysis, and briefly describes the fusion rules and evaluation indicators of infrared and visible light images, which are multi-scale analysis techniques. The development in the field of image fusion provides a clear direction.
Introduction:
Image fusion aims to combine the advantages of different sensors, fuse multiple images of the same target acquired at the same time, and output an image that can be better analyzed and processed by a computer. At present, image fusion technology is widely used in military, remote sensing, medical and other fields. Infrared and visible light image fusion is a research direction. Infrared and visible light image fusion belong to different types of sensor image fusion. The imaging mechanism of the two sensors is different, and the scene information contained is also different: the visible light image reflects the ability of different objects to reflect visible light, and the texture details are rich, but they rely on lighting. Once the target scene has low visibility or obstruction, effective information will not be obtained; infrared The image represents the thermal radiation of the target scene and is not affected by the complex external lighting conditions, but it is more dependent on lighting. Once the visibility of the target scene is low or obstructed, effective information will not be obtained; the infrared image represents the thermal radiation of the target scene and is not affected by the complex external light. Condition influences to form a fusion image with rich target information.
According to the different processing domains in the process of infrared and visible light image fusion, it is divided into two types: spatial domain method and transform domain method. The spatial domain method directly operates and calculates the pixels of the two images, which has strong real-time performance, but cannot fully consider the pixel points. The associated information, texture details are seriously lost; the transform domain method starts from the sparse representation of the image, which has spectral and spatial consistency, and is the current mainstream research direction of image fusion.
1 Multi-scale analysis method
The basic idea of image fusion based on multi-scale analysis is to perform the same method of multi-scale decomposition on the two source images of infrared and visible light, and then perform the fusion operation on the decomposed high-frequency and low-frequency component coefficients according to certain fusion rules. The fusion image is obtained by the inverse scale transformation. At present, common multi-scale analysis methods include wavelet transform, non-subsampled Contourlet transform, Shearlet transform, Tetrolet transform, etc. These fusion methods based on transform domain have become the development direction of infrared and visible light image fusion.
1.1 Wavelet transformWavelet
transform has the characteristics of excellent variable time-frequency domain resolution, direction selectivity, and small amount of analysis data. In the image fusion process, the source image is first subjected to wavelet decomposition to obtain low-frequency image information, as well as vertical and horizontal The sub-image information in the three diagonal and diagonal directions uses different fusion rules to create images at different frequency components, and reconstructs the fused image through inverse wavelet transform. Wavelet transform can reflect the singularity of image points, but it cannot optimally express the boundary and line feature information of the image. Therefore, the fused image has obvious edge blur and block phenomenon.
1.2 Non-subsampled Contourlet TransformThe
non-subsampled Contourlet transform is a representation method that obtains multi-direction, multi-resolution and anisotropy through an iterative method. It consists of a non-subsampled tower filter and a non-subsampled directional filter. The former decomposes the image into low-frequency and high-frequency sub-band images, and the latter performs directional filtering on high-frequency sub-images. There is no down-sampling process in the decomposition and reconstruction process, and it has the characteristics of translation invariance and is well eliminated. The pseudo Gibbs effect is protected, and the edge texture and contour structure information of the image is protected.
1.3 Shearlet transformation The
Shearlet transformation constructs an anisotropic basis function by performing affine transformations such as translation, shearing, and expansion of the basic function. Its support interval forms a wedge-shaped structure that can change with the scale. Therefore, the Shearlet transformation is no longer affected by the basis function. The constraints of the size and the number of directions can obtain more directional information, and the image edge detail positioning ability is better.At the same time, it is different from the inverse synthesis method of the directional filter bank of the non-subsampled Contourlet transform. The image reconstruction of the Shearlet transform is just addition. Processing, the algorithm efficiency is improved.
1.4 Tetrolet transform
Tetrolet transform is a Haar wavelet transform based on four imposition concepts, which can adaptively select the best template for sparse processing according to the geometric structure of the source image itself, and obtain better image quality. The decomposition process is to first divide the source image into a number of 4×4 image blocks, perform Tetrolet transformation on each block to obtain 4 low-frequency coefficients and 12 high-frequency coefficients, and then arrange the low-frequency coefficients according to certain rules to generate and original For a new image with the same image size, continue the 4×4 block operation on the image and perform the Tetrolet transformation until the decomposition level required by the fusion is completed. Tetrolet transform has more directionality and selectivity, and is more suitable for the structure of the image itself, and the spatial details and edge texture information of the image are better preserved.
2 Fusion rule
Image fusion is carried out through multi-scale analysis method. The two images that are decomposed will generate different image coefficient matrices. The high-frequency coefficient contains the characteristic detail information of the image, and the low-frequency coefficient contains the contour structure information of the image. For related operations, it is necessary to select appropriate fusion rules, which can capture the unique information of each image while retaining the corresponding general information. Fusion rules directly affect the quality and effect of image fusion. Currently, it is mainly divided into pixel-based and region-based methods.
2.1 Pixel-based
method According to the characteristics of the image pixel matrix arrangement, this method compares the pixels at the same position of each source image, and selects the largest absolute value or the weighted average value as the pixel value at the corresponding position of the final fused image. This method is simple to operate, easy to implement, and fast in fusion, but it ignores the correlation between pixels, lacks relevant considerations between single pixels and surrounding pixels, and cannot describe the local features of the image well. At the same time, it is aimed at infrared The image and the visible light image have the characteristic of large difference in gray value. This pixel-based fusion rule has a mediocre effect.
2.2 Region-based method
This method treats each pixel as a part of the region, and finds the features of the image region through methods such as regional energy and regional variance, and compares these features according to certain standards, and finally meets the requirements. The area is used as the fused image area. Different regional feature methods reflect the image contour and texture information differently. For example, a large regional gradient value indicates that the image content of this region has changed greatly; a large regional energy value indicates that the image detail information of this region is more. At present, the fusion of infrared and visible light images is mostly based on the method of regional fusion rules.
3 Evaluation
of fusion performance Evaluation of the effect of image fusion can evaluate the pros and cons of fusion algorithms. Fusion image evaluation is an important part of the infrared and visible image fusion system. Currently, the commonly used evaluation methods are subjective and objective evaluation methods. The subjective evaluation method is to use the human eyes to directly observe the fusion image and judge the image quality qualitatively. Observers use experience or subjective preference to give evaluation opinions based on established evaluation rules such as image distortion and detail display. This method is simple and direct, and has good recognizing power for information such as space and shape. It can judge whether the edges are complete and the shape is distorted. It is more suitable for infrared and visible image fusion evaluation. But this method is very subjective, especially when the image difference is small, the conclusion is not reliable. Therefore, it is necessary to combine the objective evaluation method to quantitatively analyze the fusion quality, judge whether the fusion effect meets the standard, and ensure the scientificity of the fusion result.
The objective evaluation method is based on theories such as information theory, image gradient, and human vision. According to different conditions, it is mainly divided into the following three types of evaluation indicators: (1) The method of evaluation based on the degree of retention of the source image information in the fusion image, mainly including Information entropy, cross entropy, interactive information volume, etc. (2) The evaluation method based on the display effect of the spatial detail information of the fusion image, mainly including the average gradient and the spatial frequency. (3) The evaluation method is based on the degree of deviation between the fusion image and the source image's spectral information, mainly including correlation coefficient, deviation index and so on.