Article directory
Included in ECCV 2020, code address:https://github.com/zzh-tech/Animation-from-Blur
Research Background
The main solution to the traditional image blur problem is1-to-1 mapping, that is, input a blurred image and output the corresponding one Clear image. However, this method ignores the effective timing information contained in blur. For example, motion blur can be understood asthe average change of the scene recorded within the short time of camera exposure< a i=4>. Therefore, recent work has begun to pay attention to a more challenging problem, that is, obtaining a set of clear image sequences (short videos) througha single motion blurred image, which is equivalent to the inverse process of motion blur formation. This problem is also called blur decomposition or blur interpolation.
The author analyzed the "fuzzy decomposition" problem and found that for each fuzzy area (such as the left and right hands, left and right feet, etc. in the picture below), "< a i=8>forward motion"and"backward motion"Two possible solutions ( forward or backward motion), as the motion blur area in the image increases, the number of potential solutions will grow exponentially, which ismotion ambiguityProblem(motion ambiguity). However, current methods do not take motion ambiguity into account, which causes the model to only predict one solution among many solutions, bringing instability to the training process. If this problem is not addressed, it can lead to low model diversity and poor quality generation.
Solutions
In order to solve the problem of ambiguity, the author introduces motion guidance representation (motion guidance), which is 2D optical flow in four directions A rough quantitative representation (only direction without amplitude), as shown in the figure below (different colors represent different directions of motion). With motion guidance, it becomes a one-to-one mapping problem, and the model can generate an image sequence solution with a specific direction and no ambiguity.
The author further proposes a two-stage fuzzy decomposition network, as shown in the figure below. The first stage generates a preliminary image sequence by inputting blurred images and motion guidance, and the second stage further refines visual details through residuals.
Motion guidance is required in both the training and inference stages of the model. The author proposed three different ways to generate motion guidance corresponding to blurred images. The first wayuses the cVAE-GAN network to learn the motion guidance distribution corresponding to the blurred image, which can be obtained by sampling different motion guidance during inference. Generate different solutions; the second way is that if the input is a blurred video sequence, then you can directly perform optical flow estimation and perform quantization to get motion guidance; the third way is to use the user to draw a rough motion guidance for decomposition.
Experimental effect
The figure below shows the comparison between this method and other methods, as well as the generation effect under different motion guidance.
think
This article solves the ambiguity problem in multiple solution problems. The solution is also very simple: since there is ambiguity in the generation process, then give an additional guidance to limit it and turn it into a One-to-one questions. This idea is also worth learning from in other fields, mainly finding the key factors that generate ambiguity and proposing corresponding wayto solve it.
References
[1] Focus on the new challenge of image/video deblurring: Blur Decomposition/Interpolation