CVPR2020丨SPSR: Super-resolution method of structure preservation based on gradient guidance

Click on " Maiwei AI Research Institute " above and select the " Star ★ " public account

Heavy dry goods, delivered immediately

Author | Alan

Authorized to reprint from | https://zhuanlan.zhihu.com/p/121721537

CVPR2020：Structure-Preserving Super Resolution with Gradient Guidance

Paper: https://arxiv.org/pdf/2003.13081.pdf
Code: https://github.com/Maclory/SPSR

We know that structural information is very important in super-resolution problems. There are already many perceptual-driven methods (SRGAN, ESRGAN, EnhanceNet) that use GAN Loss and Perceptual Loss to generate more realistic and detailed super-division images. However, there are always some structural deformations in these restored images that we do not want it to appear, as shown in Figure 1 below.

Although the contour edges of the image generated by RCAN using only L1 Loss are relatively regular, the overall image is blurry. The images generated by SRGAN, ESRGAN, NatSR, etc. have obvious structural distortions.

Figure 1. Partial image detail comparison

Therefore, the paper proposes a structure-preserving super-resolution method SPSR to alleviate the structural deformation while retaining the advantages of the GAN method to generate rich texture details. In particular, the paper uses image gradient information to guide image restoration.

How to guide image restoration through gradient information?

There are two main points:

1. SPSR uses an additional gradient branch network to generate a high-resolution gradient map as an additional structural prior.

2. SPSR introduces a new gradient loss and imposes secondary constraints on super-resolution images.

Therefore, gradient information and gradient Loss can further help generate the network to pay more attention to the geometric structure of the image. In addition, the SPSR method is also applicable to other SR model methods.

It can be seen from Figure 1 that SPSR can not only generate clearer results than RCAN, but also retain a finer geometric structure.

图 2. Overall framework of SPSR

Network structure

1. Gradient Map

First, SPSR extracts the LR gradient map from the LR image, and obtains it by calculating the difference between adjacent pixels:

The code based on the formula is as follows (please refer to the source code for specific implementation):

import torch
import torch.nn as nn


class Gradient_Map(nn.Module):
    def __init__(self):
        super(Gradient_Map,self).__init__()
        self.pad =  nn.ReplicationPad2d((1,0,1,0))
        
    def get_gray(self,x):
        ''' 
        Convert image to its gray one.
        '''
        gray_coeffs = [65.738, 129.057, 25.064]
        convert = x.new_tensor(gray_coeffs).view(1, 3, 1, 1) / 256
        x_gray = x.mul(convert).sum(dim=1)
        return x_gray.unsqueeze(1)

    def forward(self,x):
        x = self.pad(x)
        x = self.get_gray(x)
        h = x.size()[2]
        w = x.size()[3]
        I_x = torch.pow((x[:,:,1:,1:]-x[:,:,:h-1,1:]),2)
        I_y = torch.pow((x[:,:,1:,1:]-x[:,:,1:,:w-1]),2)
        M_I = torch.pow(I_x+I_y,0.5)
        return M_I

2. Gradient branch

As can be seen from Figure 2, the modules in the gradient branch network will first merge the output features of several intermediate layers from the SR branch network. This is because the intermediate layer of the SR branch network contains rich structural information, which is useful for the recovery of the gradient map. The key role. Secondly, at the end of the gradient branch network, the feature information of the gradient map will be merged into the SR branch network to guide the SR reconstruction process.

3.SR branch network SR branch

There are two parts to the SR branch network:

The first part: adopts the same network structure as ESRGAN, a total of 23 RRDB modules. And the output of the 5th, 10th, 15th, and 20th RRDB modules are used to merge into the gradient branch network.

The second part: connect the SR gradient feature map obtained from the gradient branch network, and then use an RRDB module + a convolutional layer to generate the final SR feature.

PS: The paper did not introduce too much about the implementation details of the network model, so the specific implementation still needs to wait for the author to open the source code.

Objective function and loss

1. Traditional Loss:

First used L1 Loss and Perceptual Loss:

It also introduces GAN Loss:

2. Gradient Loss:

In order to illustrate the role of the gradient Loss, the author of the paper gives an example in the 1-D case. It can be seen that if the model is optimized in the image space only through the L1 loss, when the input test sequence is given, the usually obtained SR sequence is shown in Figure 3(b), and the GT sequence is shown in Figure 3(a ) As shown.

The model cannot recover the sharp edges because the model tends to give the median value of all HR sequence statistics from the training data. So in this case, if you calculate and display the gradient values of the two sequences, you can observe that the SR gradient is flat and the peak is low, while the HR gradient is a sharp peak. The gradient distribution of the two is obviously different.

Therefore, by introducing the gradient Loss, the model can better learn image features from the gradient space. It helps the model focus on the image neighborhood, so that the local intensity of image sharpness can be inferred more appropriately. Therefore, when the image gradient information, as shown in Figure 3(f), is captured by the model, then the SR method can better guide the gradient information to avoid excessive blur or excessive sharp restoration, thereby generating more Realistic super-resolution images.

Figure 3. 1-D example of gradient information.

The specific form of the gradient Loss is as follows:

It is used to measure the absolute error between the SR gradient feature map generated in the SPSR network and the true gradient feature map.

SPSR designed an additional gradient discriminant network to optimize the following loss functions. The gradient feature map is the gradient feature map of the SR image:

The gradient discriminant network can also supervise the SR image generation network through adversarial learning:

It is worth noting that the gradient extraction operator is differentiable, so the model can be directly trained end-to-end.

The overall optimization objective function is:

Among them, are the weight parameters of different Loss, which are set to respectively in the paper .

Experimental results and analysis

In order to illustrate the effectiveness of SPSR, the paper describes the quantitative and qualitative comparison.

First, from a quantitative point of view, the paper selects SFTGAN, SRGAN, ESRGAN, NatSR and other GAN network-based SR models for comparison with SPSR. Evaluation indicators used four indicators: PI, PSNR, SSIM, and LPIPS. It can be seen that SPSR has achieved the best performance in LPIPS and PI indicators.

Table 1. Performance evaluation on Benchmark

Then through the visual comparison of the generated images, it can be seen that there are obvious deformations in the geometric structure of the local regions of the images generated by other methods, and the results generated by SPSR are more natural and real. In addition, the SPSR image is purer without obvious artifacts and distortion. The above also fully illustrates the effectiveness of the SPSR method.

Figure 4. Comparison of local details

to sum up

The paper proposes a structure-preserving super-resolution method SPSR based on gradient guidance, which alleviates the problem of geometric deformation that is common in the SR results generated by the perceptual driving method. Not only is the gradient branch network designed to generate the SR gradient feature map, but also the gradient Loss is introduced to make further constraints. This also provides a new idea for the subsequent study of super-resolution methods, using different image prior information as constraints to guide the training of the SR model, so as to promote the SR model to generate more natural and realistic images.

Recommended reading

(Click the title to jump to read)

Now it is ⛴Maiwei AI Research Institute⛴

Click " Read the original text " to view more historical content, and like to click " Looking at ❀ "!