[Space-time Fusion: Improving MRA]

Multiresolution Analysis Pansharpening Based on Variation Factor for Multispectral and Panchromatic Images From Different Times

(Multi-resolution analysis of multispectral and panchromatic images based on variation factors)
Most pan-sharpening methods combine the original low-resolution multispectral (MS) image and the high-resolution panchromatic (PAN) image acquired simultaneously on the same area mix together. Multi-resolution analysis (MRA) has become one of the important categories of pan-sharpening methods due to its good robustness. However, when only MS and PAN images at different times can be provided, the fusion results of existing MRA methods are often unsatisfactory due to the inability to effectively analyze the multi-time deviations between MS and PAN images at different times . To address this problem, a MRA pan-sharpening method for MS and PAN images based on variation factors is proposed. First , an MRA pan-sharpening method based on a dual-scale regression model was established , and then a variation factor was introduced to effectively analyze multiple time differences using the alternating direction method of multipliers (ADMM) to obtain the final fusion result. Experiments on synthetic and real datasets show that the proposed method achieves significant performance improvements compared to traditional pansharpening methods as well as state-of-the-art MRA methods. Visual comparison shows that the introduction of the change factor improves the multi-temporal misalignment compensation of ground objects and promotes the application of pan-sharpening of MS and PAN images acquired at different times.

INTRODUCTION

Multispectral remote sensing data fusion refers to the spatial registration of remote sensing image data, including multi-temporal registration, mass spectrum registration, multi-sensor registration, multi-platform registration and multi-resolution registration. Obtain images from the same area, and then use a certain algorithm to organically combine the advantages of each image to generate new data. Multi-source remote sensing data technologies include resolution enhancement, feature extraction, cloud removal, classification, super-resolution and multi-temporal data fusion. Fusion of MS and PAN images, known as pansharpening, can obtain high-resolution sharpening data. As high-resolution imagery increases the availability of commercial products, there is a growing demand for pan-sharpened data from satellite remote sensing systems. In addition, pansharpening is an important preliminary step in the processing of change detection, target recognition, land cover classification, visual image analysis, and scene interpretation of remote sensing images. Another common fusion technology, namely space-time fusion, is a method of mixing fine spatial resolution data with coarse temporal resolution data (such as Landsat), fine temporal resolution data and coarse spatial resolution data (such as MODIS) to form a temporal Sequential image technology. However, spatiotemporal fusion usually requires two coarse/fine image pairs to estimate the temporal change rate of each class separately, assuming that the change rate is stable over a period of time. In this sense, pan-sharpening can be more accurately viewed as a spatial-spectral fusion technology, while classical spatio-temporal fusion can be viewed as a multi-temporal fusion technology. The two fusion technologies have different purposes and application objects.
In general, pansharpening methods can be divided into four major categories: 1) component substitution (CS); 2) viewing; 3) variational optimization (VO); 4) deep learning (DL). In the CS method, the geometric details to be injected into the interpolated MS band are extracted from the PAN image through spectral transformation of MS pixels. This category includes intensity hue saturation, principal component analysis, Gram-Schmidt, and context-adaptive Gram-Schmidt (C-GSA). CS-based methods can work well when there is a strong correlation between high-resolution PAN images and low-resolution MS images, but they cannot account for local differences caused by spectral mismatch. Therefore, the fused image will have significant spectral distortion. The VO method relies on the solution of the optimization problem, including methods such as total variation, sparse representation, Bayesian method, and reduced rank (RR). However, the time complexity of the VO method is very high and it is difficult to achieve real-time performance. In the past decade, DL-based methods have attracted increasing attention, including convolutional neural networks, pan-sharpening neural networks, and pan-sharpening neural networks that enhance the input by including several nonlinear radiation index maps. , and transformer-based pan-sharpening. Although DL-based methods can achieve good performance, they usually require large amounts of labeled training data. To address this problem, Xu et al. recently proposed an iterative network unsupervised learning method based on spectral and texture loss-constrained generative adversarial networks, which does not require labeled datasets for training. In addition, Qu et al. proposed an unsupervised learning method based on the self-attention mechanism, and Liu et al. proposed a spatio-temporal attention fusion method. However, it is still not suitable for the fusion of MS and PAN images from different satellites and times, and its feasibility as a solution to the pansharpening problem remains to be fully tested and verified.
MRA-based methods retain the spectral information of the original mass spectrometry data set regardless of the date or instrument in which the PAN was obtained, and therefore have become one of the mainstream pan-sharpening methods. MRA categories include additive wavelet luminance scaling (AWLP)] and morphological filters (MFs). In particular, Aiazzi et al. used the mass spectrometry sensor MTF-based glp (MTF-glp) to perform the analysis step. On the basis of this method, context-based decision-making MTF-GLP (MTF-GLP-cbd) and comprehensive regression-based MTF-GLP (MTF-GLP-fs) ​​are proposed. In addition, related studies have shown that MTF-GLP-based methods can be improved using the HPM injection protocol [MTF-GLP vs. HPM injection (MTF-GLP-HPM)] model. In addition, based on the MTFGLP-HPM model, post-processing [MTF-GLP-HPM with post-processing (MTF-GLP-HPM-pp)] and multiple linear regression [MTF-GLP-HPM with multiple linear regression (MTF-GLP -HPM- r)] have also been proposed one after another. In addition, in order to utilize richer scale information, literature [46] proposed a dual-scale regression model [MTFGLP-HPM with dual-scale regression (MRA-DS)] to obtain the best fusion results.
However, physical limitations of spatial resolution and spectral resolution in terms of (wrt) signal-to-noise ratio, limited transmission bandwidth, and processing capabilities hinder the simultaneous collection of high spatial and high spectral resolution data using remote sensing instruments. Therefore, it is sometimes difficult to obtain MS images and high-resolution PAN images of the same area at the same time. When only MS and PAN images acquired at different times can be provided, the fusion results of the pansharpening method are often unsatisfactory due to the multi-temporal misalignment of the ground object images acquired at different times. Although relevant studies have shown that MRA classification can better compensate for multi-time axis misalignment compared with other pansharpening classifications, these MRA methods are only directly used to fuse MS and PAN images at different times and cannot effectively alleviate multi-time axis misalignment. Influence.
In order to solve the multi-period misalignment problem, this paper proposes an MRA pan-sharp technology based on variation factor (MRA-VF). Experimental results show that this method is better than the traditional pan-sharpening method. To summarize, the contributions of this article are as follows:
1) Based on the author's knowledge in MRA classification methods, the MRA-VF proposed in this article is the first method to analyze the multi-temporal deviation of MS and PAN images acquired at different times on ground objects. Therefore, the main purpose of this article is the improvement of the MRA category , and the study of other categories is beyond the scope of this article.
2) In the MRA-VF proposed in this article, when the difference between the two data sets is large, the information of the PAN image will be effectively extracted and suppressed through alternating optimization to avoid the introduction of erroneous information.
3) When MS and PAN images are acquired at different times, the variation factor is applied as a general model to other methods of the MTF-GLP-HPM type to improve the fusion results.

METHODOLOGY

The flow chart of MRA-VF is shown in Figure 1. First, an MRA model based on dual-scale regression was established, and the variation factor was introduced; finally, through a combination of alternating direction multiplication, the variation factor was used to effectively analyze multiple time differences, and the final fusion result was obtained.
Let $P^{HR}$∈$R^{M\times N}$ is a high-resolution PAN image, where M is the number of rows of the PAN image, and N is the number of columns of the PAN image. Low-resolution MS images are represented as$M^{LR}$ = { $M^{LR}$}b=1，…，B∈$R^{(M/S)\times (N/S)\times B}$ , where B is the number of spectral bands, S is the ratio scale of P HR and MLR, and MLR B is the Bth spectral band. The superscript indicates the spatial resolution of the image, that is, LR and HR represent low-resolution and high-resolution images respectively.

Dual-Scale Regression Model

Since our previous work, the dual-scale regression model (MRA-DS) performed better than many other MRA methods, MRA-DS has become one of the state-of-the-art MRA methods. Therefore, MRA-DS is selected as the basic mathematical model of the proposed MRA-VF.
First, change $M^{LR}$ interpolated to$P^{HR}$ size, get ^$M^{LR}$ = {^$M^{LR}$}b=1，…， b∈$R^{M\times N\times B}$ . Then use MTF-GLP[40] to$P^{HR}$ is processed to obtain a low-pass version of the low-resolution PAN image, that is,$P^{LR}$ .${}^{\_}$Next, use the injection coefficient g to control the difference in information injection, as shown in the following formula, and obtain the final pansharpening result$M^{HR}$ = { $M^{HR}$}b=1，…， b∈$R^{M\times N\times B}$ :
where g_bis the injection coefficient of the second spectral band of the MS image. Furthermore, HPM injection scheme and scale regression are used to improve the performance of pansharpening [43]. Based on the HPM injection scheme, (1) is rewritten as According to
the scale regression method of [45], (2) can be rewritten as
where E(X) represents the mean value of image X, and the injection coefficient gb is used for iterative operation.
In MRA-DS, in order to enrich the scale information and improve the final fusion result, a dual-scale regression method is proposed, which combines$P^{LR}$与$M^{}$Fine-scale information of covariance between${}^H$${}^R$ _b$P^{LR}$与$M^{}$The coarse-scale information of the covariance between${}^H$${}^R$ _bTherefore, gb is defined as
where cov(X, Y) is the covariance of images X and Y, var(X) is the sample variance of image X, i is the number of iterations, and µ is an adjustable parameter. The appendix details the validity of the biscale regression.
The key difference between the MRA-VF proposed in this paper and the MRA-DS in our previous work is that MRA-DS does not adapt well to MS and PAN images at different times. Therefore, the key innovation is that based on the authors' knowledge of MRA classification methods, the MRA-VF proposed in this paper analyzes for the first time the multi-temporal misalignment of ground objects between MS and PAN images acquired at different times.

Variation Factor

In order to analyze the multi-temporal phase dislocation of ground objects and apply pansharpening technology to MS and PAN images acquired at different times, the variation factor θ∈ RM × NR^{M×N} is introduced into the dual-scale regression model.$R^{M\times N}$ , the proposed MRA-VF is obtained. Rewrite equation (3) as

where ⊙ represents the Hadamard product. This model represents multi-temporal misalignments caused by seasonal or anthropogenic changes in ground objects.
The next problem is to find θ. To this end, the multiple time differences between MS images and PAN images acquired at different times are connected, and the following equation is established:
But in practical applications, MS images are three-dimensional data and PAN images are two-dimensional data. In order to express (6), the dimensionality of the MS image needs to be decomposed and reduced to facilitate processing in alternating direction multiplier (ADMM) iterations. Inspired by pansharpening, the singular value decomposition (SVD) method can be used to factorize the three-dimensional MHR to obtain the two-dimensional MHR. We use SVD to reduce the dimensionality of MS images because keeping MS in a two-dimensional space is more convenient for subsequent operations of the ADMM algorithm. If we use histogram matching to expand the dimensionality of PAN images, we need to consider tensors to process ADMM, thus increasing the complexity of the algorithm

Optimization of Variation Factor by ADMM

It can be seen from observation (5) that when the optimal variation factor θ ^∗ is obtained , the proposed MRA-VF can effectively analyze the multi-temporal misalignment of ground objects between MS and PAN images acquired at different times and obtain good Fusion results. Therefore, an optimization model is established for (6) to realize the optimization of the variation factor θ, as shown in the following formula:
The first regularization term can control the spectral variability by constraining θ to be close to unit. The second regularization term enhances the smoothness of the spectrum by establishing a differential operator Hl. Parameters µ1 and µ2 balance the contributions of different regularization terms to the cost function.
Due to the advantages of distributed optimization, the optimization model in (8) can be solved using the ADMM framework. Specifically, the cost function in (8) can be expressed as
ADMM can be used to decompose problem (8) into simpler problems and solve it iteratively [56]. Specifically, the subproblem can be expressed as
Thus, (8) can be updated by minimizing the following problem: The
augmented Lagrangian associated with (10) is given by
Secondly, the optimization of the variables φ and θ is introduced process.

1. Optimization wrt ϑ: The optimization problem for φ can be written as
   This expression can be rewritten for each pixel individually as Taking
   the derivative of each pixel in equation (16) and setting it to 0, we can get:
   2) Optimizing wrt θ: This The optimization problem is equivalent to equation (13) when c = 0 and can be written
   as For simplicity, the non-negative constraints are initially ignored. Derive θ and set it to 0. The result is
   that due to multi-period misalignment, the large differences between MS and PAN images need to be ignored to avoid introducing too much erroneous information to the MS image. In order to solve this problem, a threshold δ needs to be set, in units of U. For each U obtained, its maximum value U _m and different values ​​δ∈[0,1] are determined according to different data sets. If θ is larger than δ * U _m , then θ is set to 0 to remove erroneous information in the PAN image.
   After obtaining the optimal variation factor θ *, substitute it into equation (5) to obtain the pan-sharpening result. Algorithm 1 gives the process of the proposed MRA-VF