[Multi-scale enhancement network: super-resolution]

Remote Sensing Image Super-Resolution via Multiscale Enhancement Network

(Remote sensing image super-resolution based on multi-scale enhancement network)

In recent years, remote sensing images have attracted widespread attention due to their special value. However, satellite remote sensing images are usually of low resolution, so it is difficult to infer high-frequency details from remote sensing images compared with ordinary digital images, which means that they cannot meet the needs of some downstream tasks. We propose a multi-scale enhancement network (MEN), which uses multi-scale features of remote sensing images to improve the reconstruction ability of the network. Specifically, the network extracts coarse features of LR remote sensing images using convolutional layers. These features are then fed to the multi-scale enhancement module (MEM) proposed by this network, which uses a combination of convolutional layers with multiple convolution kernel sizes to refine the extraction of multi-scale features , and finally, generated by the reconstruction module Final reconstructed image.

INTRODUCTION

Image super-resolution (SR) is an important technology in computer vision to improve image and video resolution. Remote sensing images can observe Earth objects from outer space and provide rich information for monitoring the Earth's surface. In recent years, remote sensing images have been widely used in environmental monitoring, resource exploration, land cover classification and other fields. However, due to the influence of hardware and environmental factors, the quality and resolution of the obtained remote sensing images are low. Therefore, how to improve the resolution and image quality of remote sensing images has become a research hotspot for many researchers.
Recently, due to the rapid development of convolutional neural networks (CNN), SR methods based on deep learning have gradually entered the mainstream. Shi et al. constructed a sub-pixel CNN and introduced sub-pixel convolution into image upsampling to improve the image reconstruction effect. Lei et al. designed a new multi-branch structure to enhance the reconstruction performance of the reconstruction network by extracting local and global features of remote sensing images. Jiang et al. adopted a dense residual reconstruction method to enhance the reconstruction ability of remote sensing image features. Xu et al. proposed a SR method based on deep memory networks to enhance the reconstruction performance of the network through local and global memory connections. Dong et al. designed a multi-sensory attention reconstruction network that uses multi-sensory learning and multi-level information to reconstruct images. Lu et al. proposed an SR reconstruction network that utilizes the multi-scale characteristics of remote sensing images to effectively reconstruct the high-frequency information of remote sensing images. Although the above remote sensing image network provides better reconstruction results, the network reconstructed by deepening SR often introduces redundancy of feature information. In addition, due to complex factors such as remote sensing image degradation and content diversity, multi-scale information extraction of remote sensing images is the key detail information of remote sensing image SR.
In order to solve the above problems in remote sensing image reconstruction, we propose a new multi-scale enhancement network (MEN) for remote sensing image reconstruction. Specifically, this network extracts coarse features of LR remote sensing images through convolutional layers. These features are then input into the multi-scale enhancement module (MEM) proposed in this network, which uses a combination of convolutional layers with multiple convolution kernel sizes and refines the extraction of multi-scale features of remote sensing images. Then residual hopping is used to achieve effective fusion of features, so that MEN can effectively extract multi-scale detailed information of remote sensing images to the maximum extent.

In summary, our contributions are summarized as follows:
1) We design a new remote sensing reconstruction network MEN, which can not only adaptively detect features of remote sensing images, but also achieve effective fusion of multi-scale features of remote sensing images. Relevant experiments have proven the superiority of the MEN reconstruction network, and subjective and objective comparisons can outperform other state-of-the-art SR methods.
2) We adopt a new MEM in the SR reconstruction network, which is mainly composed of convolutional layers of different scales, so that the reconstruction network can better extract multi-scale details of remote sensing images. The concat layer in the module can reduce redundant features and improve the utilization of feature information.

PROPOSED METHOD

Network Architecture

The ultimate goal of our MEN is to develop a mapping function F that can reconstruct SR remote sensing images $I^{SR}$ from input image$I^{LR}$ .${}^{\_}$In our study, the original high-resolution (HR) remote sensing image IHR is used to create a low-resolution (LR) remote sensing image$I^{LR}$ through bicubic operations. ILRI^{LR}discussed$I^{LR}$ ∈$R^{H\times W\times CsumIHRI}$ ^$I^{HR}$，$I^{SR}$∈$R^{rH\times rW\times C}$ , where C represents the number of channels of the image, and r represents the magnification size.
Learn$I^{LR}$和$I^{}$The end-to-end mapping function F between${}^{HR}$${}^{and\; R\; is\; the\; ultimate\; goal\; of\; this\; research.}$The training data set is specified as { ILRI^{LR} $I^{LR}$_i，$I^{HR}$ _i}^N_i=1, and solves the following problem:
In recent years, the mean square error and l₂functions have become popular in image reconstruction optimization.
Despite the fact that these techniques can achieve high objective evaluation metrics, remote sensing image reconstruction using such loss functions often produces overly smooth textures. We find that the improvement in remote sensing reconstruction performance from these loss functions is subtle. Finally, our MEN network uses the l₁function to avoid introducing specialized training skills and reduce network computation. Therefore, we will$I^{SR}$ is defined as
The structure of the proposed method is shown in Figure 1. In order to stabilize network training such as gradient explosion, we use residual skip connections for feature fusion of the network. This model takes unprocessed LR remote sensing images as input, samples them into HR space through residual jump connections, and outputs reconstructed HR remote sensing images.

Multiscale Enhancement Module

We re-evaluate basic convolutional blocks for feature extraction from remote sensing images in single-image SR tasks, which are motivated by recent multi-scale feature learning. The proposed MEM consists of three sub-networks with convolutional kernels of various scales, unlike many MSRN-style blocks. The developed method makes it possible to share extracted feature data, using various convolution kernels, to obtain multi-scale information. According to Figure 1, the lower left figure depicts the specific MEM structure, and the lower right figure shows three subnetworks of different scales of the convolution kernel.
We first define the rough features of the remote sensing image input to MEM
as
After the fused features in each MEM module are nonlinearized, the features are added through residual connections and then sent to the next MEM module. Finally, the rough remote sensing image feature X is added to realize the interaction of features of different scales.