超分算法ESPCN：《Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel》

1. Overview of ESPCN

"Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel" proposes a new upsampling method , which has a good improvement in the calculation speed and reconstruction effect of SR (super-resolution) tasks .

The article introduces an SR algorithm—ESPCN—that has improved reconstruction expressiveness and computational efficiency (reconstruction speed, computational resource consumption) on previous algorithms (SRCNN, Bicubic) .
SRCNN first performs Bicubic interpolation on the input image, and then performs feature extraction. This method is equivalent to directly super-scoring at the HR level. The author proves that this method is a suboptimal strategy and will bring about an increase in computational complexity.

In response to this problem, the author proposed the ESPCN structure:

1. This is a feature extraction directly on the input LR image .
2. One is introduced in the network 亚像素卷积层. This layer is usually the last layer of the network. It takes the feature map after feature extraction as input and learns an upsampling filter to achieve LR→SR reconstruction.
3. Convolving the LR image directly replaces the bicubic preprocessing part in the SRCNN, which directly reduces the computational complexity and improves the execution speed . The author realized the super-resolution effect in the 1080P video, and achieved the "in the title" Real-Time".
4. ESPCN has done related experiments on images and videos, which have been improved by 0.15dB and 0.39dB respectively; in addition, the execution speed of ESPCN has also surpassed the previous CNN-based series SR algorithms.

2. Detailed explanation of the paper

2.1 Introduction

The main two points of ESPCN are very important: ①It is to directly extract the features of LR LR level images, ②It is a sub-pixel convolution layer. Let’s expand it below:

1. ESPCN directly convolutes the image at the LR level to extract features, so we can use a smaller filter to integrate feature information at different levels . Compared with SRCNN, this approach not only reduces the training parameters , but also reduces the Computational complexity reduces training time.
   In addition, as stated in the DCSCN paper, when r ≥ 3, there is actually no difference between the features extracted directly from the input image and the features extracted after first enlarged by bicubic interpolation, so the interpolation will appear redundant, and Wasted computational cost.
2. The way of upsampling in SRCNN is a simple process: to interpolate the input LR image, only one filter is needed. In ESPCN, assuming that the entire network has L layers, then the $n_{L-1}$Zhang feature map, then at the L layer, we can learn a more complex way to train $n_{L-1}$A convolution kernel instead of a simple one.
   More importantly, the last layer can implicitly learn the number of channels $n_{L-1}$The ground filter, which is essentially a shuffle process, will $r^2$ feature maps are integrated, that's it亚像素卷积层.

2.2 Method

The above picture is the ESPCN network structure, and the following points will be explained:

1. The input image is an LR image, and the number of channels is C. For example, in an RGB format image, C = 3, our goal is to start from $C\times H\times The\; image\; of\; W$ becomes$C\times rH\times$Image of$r$$W.$
2. The entire network is divided into two parts: the feature extraction part , which consists of a continuous CNN network; the upsampling part , which consists of a sub-pixel convolutional layer.
3. Assuming that the entire network has a total of L layers, the first L − 1 layer is composed of convolution and nonlinear activation. The specific expression is:
   $$\begin{array}{rl}{f}^1\left({\mathbf{I}}^{LR};W_1,b_1\right)& =\varphi \left(W_1\ast {\mathbf{I}}^{LR}+b_1\right),\\ f^l\left(I^{LR};W_{1:l},b_{1:l}\right)& =\varphi \left(W_l\ast f^{l-1}\left(I^{LR}\right)+b_l\right),\end{array}$$
   where $W_l,b_l,l\in (1,L-1)$ are learnable network weights and biases, respectively. $W_l$is $n_{l-1}\times n_l\times k_l\times k_l$The two-dimensional convolution tensor of $n_l$for $The\; number\; of\; l-$ layer features,$n_0=C$, $k_l$for $The\; size\; of\; the\; l-$ layer filter. Deviation$b_l$is of length $n_l$of vectors. Nonlinear function (or activation function) $\varphi$ is applied element-wise and is fixed.
4. The feature extraction layer keeps the size of the image constant.

2.2.1 Deconvolution layer

Adding a deconvolution layer is a common way to recover resolution from max pooling and other image downsampling layers. This approach has been successfully used for visualizing layer activations [49] and generating semantic segmentations using high level features from the network [24].

It can be seen that the bicubic interpolation used in SRCNN is a special case of deconvolution layers, as described in [24, 7]. The deconvolution layer proposed in [50] can be seen as each input pixel with a stride $The\; element-wise\; score\; of\; the\; filter\; elements\; of\; r$ , and the sum to the resulting output window, also known as backwards convolution [24].

2.2.2 Efficient sub-pixel convolution layer

---

To put forward the background:
Another way to upscale LR images is to perform stride $\frac{1}{r}$Convolution of , as described in [24], this can be done simply by interpolation, perforate [27] or un-pooling from LR space to HR space [49], and then convolution with stride 1 in HR space to improve performance. These implementations increase $r^2$ because the convolution takes place in the HR space.
Therefore, the author proposes a method that also uses$\frac{1}{r}$Implicit convolutional layer with stride size but no extra computation - 亚像素卷积层(sub-pixel convolution layer).

---

The core idea of ​​​​subpixel convolution:一张图像放大r倍，就相当于每个像素都放大r倍。

The convolution process of the penultimate layer of the network 输出通道数is the feature image of r^2the original image and then periodically arranged through the sub-pixel convolution layer to obtain a reconstructed image with a size of .同样大小
(w × r , h × r)

As shown in the above figure, the 9 features framed by the red circle on the penultimate layer are arranged to form the small box on the last layer pointed by the arrow. This is the reconstruction block formed by the pixels framed in the original image through the network. These nine pixels just triple the length and width of the original pixel.
Sub-pixel convolution (Sub-pixel Convolution) does not actually have a convolution operation, but simply extracts features and then arranges them simply.

---

The principle of sub-pixel convolution

1. Why is it called sub-pixel convolution?

The sub-pixel convolution layer can be regarded as a 隐式卷积process: implicit convolution means that a filter is still used to extract information, but unlike traditional convolution operations, it is not used here 没有涉及可学习的滤波器参数以及任何乘加运算.
The process from left to right in the sub-pixel convolution process is like a $stride=\frac{1}{r}$Convolve the LR image, so intuitively it seems that the convolution generates some small pixels, and because $\frac{1}{r}\le 1$ , which means that it does the operation inside the whole pixel, which we call sub-pixel, such as the common$\frac{1}{2}$、$\frac{1}{4}$Pixels are sub-pixels.

2. How does subpixel convolution work?

In LR space, a stride is $\frac{1}{r}$The convolution with a size $k_s$And the weight spacing is $\frac{1}{r}$The filter $W_s$Will activate convolution in $W_S$different parts of . Weights that fall between pixels are not activated and do not need to be computed. The number of active modes is exactly $r^2$ . Each activation mode, depending on its position, activates at most${\lceil \frac{k_s}{r}\rceil }^2$ weights. According to the position of different sub-pixels:$mod(x,r)$， $mod(y,r)$ , of which$x,y$ is the output pixel coordinate in HR space. In this paper, we propose that at$mod(k_s,r)=$An efficient way to achieve the above operation at$0\; :$

$${\mathbf{I}}^{SR}=f^L\left({\mathbf{I}}^{LR}\right)=\mathcal{P}\mathcal{S}\left(W_L\ast f^{L-1}\left(I^{LR}\right)+b_L\right)$$

where $PS$ is a periodic transformation operator, it will$H\times W\times C\cdot r^{The\; elements\; of\; the\; 2}$ tensor are rearranged as$rH\times rW\times c-$ shaped tensor. The effect of this operation is shown in Figure 1. Mathematically, this operation can be described in the following way:

$$\mathcal{P}\mathcal{S}(T{)}_{x,y,c}=T_{\lfloor x/r\rfloor ,\lfloor y/r\rfloor ,C\cdot r\cdot \mathrm{mod}(y,r)+C\cdot mod(x,r)+c}$$

Convolution operation $W_L$The shape is $n_{L-1}\times r^{2}C\times k_L\times k_L$. Note that we did not apply non-linearity to the output of the last convolutional layer. It is easy to see that when $K_L=\frac{k_s}{r}$, and $mod(k_s,r)=0$ is equivalent to filter$W_s$Perform sub-pixel convolution.
We'll call our new layer sub-pixel convolution layer, and our network efficient sub-pixel convolutional neural network(ESPCN). The last layer directly generates an HR image from the LR feature maps, each with an upscaling filter, as shown in Figure 4.

---

Training method
Given a training set containing HR Image examples $I_n^{HR},n=\mathrm{1...}N$ , we generate the corresponding LR image$I_n^{LR},n=1,..,N$ , and then calculate the pixel-wise mean squared error (MSE) as the objective function to train our network.

$$\ell \left(W_{1:L},b_{1:L}\right)=\frac{1}{r^{2HW}\_\_}\sum _{x=1}^{rH}\sum _{x=1}^{rW}{\left({\mathbf{I}}_{x,y}^{HR}-f_{x,y}^L\left({\mathbf{I}}^{LR}\right)\right)}^2$$

3. Related implementations in pytorch

Regarding the sub-pixel convolution layer, pytorch also provides the corresponding implementation torch.nn.PixelShuffle() and torch.nn.PixelUnshuffle()
usage. For details, see [torch.nn.PixelShuffle] and [torch.nn.UnpixelShuffle]

4. Reference

https://blog.csdn.net/MR_kdcon/article/details/123837994