videoSR video super-dividing paper notes one

1. Bidirectional Recurrent Convolutional Networks for Multi-Frame Super-Resolution

In the past, subpixel motions estimation is only suitable for small motions, and at the same time, this approach is computationally intensive.
The network designed by the author includes three main points: the feedforward convolution simulates the visual spatial dependence between low-resolution frames and their high-resolution results. Cyclic convolution connects the hidden layers of consecutive frames to understand the time dependence. Conditional convolution connects the input of the previous timestamp and the current hidden layer.

Use MSE to train the network.
Data set 25YUV

2. SUPER-RESOLUTION OF COMPRESSED VIDEOS USING CONVOLUTIONAL NEURAL NETWORKS(ICIP 2016)

Use a CNN trained on the temporal and spatial dimensions of the compressed video to increase the spatial resolution. Continuous frame input is used as a motion compensation input network. The network first pre-trains on the picture.

Adaptive Motion Compensation

$$Y_{t-T}^{amc}(i,j)=(1-r(i,j\left)\right)y_t(i,j)+r(i,j)y_{t-T}^{mc}(i,j)$$
$r(i,j)$ Control convex combination,$Y_t$Is the intermediate frame, $Y_{t-T}^{mc}$Is the compensation for the previous fool,
$$r(i,j)=exp(-ke(i,j\left)\right)$$
k is a constant,$e(i,j)$ is motion compensation or registration error.

3. Building an End-to-End Spatial-Temporal Convolutional Network for Video Super-Resolution(STCN AAAI2017)

The author’s network has three components: spatial information extraction, temporal information extraction (LSTM), and reconstruction. First use the spatial component to extract the features of each picture, and then send the timing information. At this time, not only the motion information can be extracted, but also the timing changes of the color information, the patch similarity of the instance objects, and so on.

3.1 The Spatial Component

Stack a sufficiently deep nonlinear network.

3.2 The Temporal Component

Using LSTM as the main architecture, while achieving multi-scale spatial information and two-way timing information, the two-way timing operation feels like the first article understands that there is no problem here.

3.3 Rebuild the module

3.4 SRCN training

The author later discovered that using $({And}_0-X_0)$和$({And}_i-X_i)$ Replace$Y_0$和$Y_i$Better results. (Residual learning, fast convergence)

4.Real-Time Video Super-Resolution with Spatio-Temporal Networks and Motion Compensation(ICCV 2017)

The previous ESPCN (sub-pixel convolution network) was only used to process pictures. Using spatial transformer networks, two pictures can be spatially mapped, and it has been applied to optical flow feature coding. So the author proposes a spatio-temporal network that combines sub-pixel convolution and motion compensation, which can quickly and accurately process video super-division. And analyzed several fusion methods, early fusion, slow fusion and 3D convolutions.

4.1 Method

The author's input is the y channel

Sub-pixel convolution SR

Mainly the idea of ​​this article
Real-Time Single Image and Video Super-Resolution Using an Efficient Sub-Pixel Convolutional Neural Network
related notes in the ESPCN
author believes that if there is a better upsampling operation than bibubic upsampling, the network can learn.

Spatio-temporal networks

Taking the current frame as the center and the number of frames before and after as the radius, the network training adds the depth of the timing
$$I_t^{SR}=f(I_{[t-R:t+R]}^{LR};\theta )$$
So the weight filter is expanded to$d_l\times n_{l-1}\times n_l\times k_l\times k_1$, The author believes that more than one frame of images can be re-added at the same time. And explored several fusion methods.

Early fusion
folds all the time information in the first layer, and the remaining operations are the same as those in the single image SR network. This method is applied in video classification and action recognition, and it is also one of the structures proposed by VSRnet.

Slow fusion
gradually merges timing information through a layered approach. There is also timing information between some layers, until all the information is merged into a depth according to a certain fusion speed. This method has a better effect on video classification.

3D fusion
is a variant of slow fusion. If the same layer of slow fusion shares parameters in the temporal dimension, it becomes a 3D convolution.

Spatial transformer motion compensation

First introduce motion compensation between two frames. Find the best (pixel-wise dense) optical flow representation between the two frames.
First get the predicted optical flow, ${\Delta }_{t+1}$Get $x$ ,yyThe offset at$y$ . So a compensation image can be expressed as$I_{t+1}^{{}^{\prime }}(x,and)=I(I_{t+1}(x+{\Delta }_{t+1}x,Y+{\Delta }_{t+1}y))$,$I(.)$ Is a bilinear interpolation operation.

The first is a 4 times sparse optical flow estimation obtained from the early fusion of the two frames, and then down-sampling by convolution with stride of 2, and sub-pixel convolution to estimate the optical flow, the resulting${\Delta }_{t+1}^c$Used on the target frame to produce $I_{t+1}^{{}^{\prime }c}$, Next is a fine flow estimation module. Get the final motion compensation frame $I_{t+1}^{{}^{\prime }}$. The activation function uses tanh.
The author used Huber loss when training this module

After that, the SR module can be added behind the network. Since these two modules are differentiable, end-to-end training can be achieved.

5.Detail-revealing Deep Video Super-resolution

Report video portal
Related notes
Note 1
Note 2
Sub-pixel motion compensation