Old Photo Restoration via Deep Latent Space Translation

Paper
microsoft/Bringing-Old-Photos-Back-to-Life
"Recovering old photos through deep latent space transformation"

• Abstract
  Using deep learning methods to restore old photos that have suffered severe degradation. This paper proposes a triple area transformation network that uses real photos and a large number of synthetic image pairs. That is, train two variant autoencoders to convert old photos and clean photos into two potential spaces respectively. And use the synthetic paired data to learn the transformation of the two latent spaces. Since the regional gap is closed in the compact potential space, this conversion can be well extended to real search. In addition, in order to solve the multiple degradation problems mixed in an old photo, the designed global branch has non-local blocks for structured defects (such as scratches and dust spots), and the local branch is for unstructured defects (such as noise spots and blurs). degree). These two branches are merged in the latent space, thereby improving the ability to restore old photos from multiple defects. In addition, the face optimization network is also used to restore the fine details of the face in the old photos, so as to finally generate photos with the quality of Zeng Qian's perception.
  

• Network structure
  
  

• 潜在空间变换
  
  $$r_R\to and=G_{and}\circ T_{With}\circ E_R(r)$$
  $$r\in R,x\in X,and\in Y$$
  $$E_R:R\to {WITH}_R,G_R:{WITH}_R\to R$$
  $$E_X:X\to {WITH}_X,G_X:{WITH}_X\to X$$
  $$E_{And}:AND\to {WITH}_{And},G_{And}:{WITH}_{And}\to Y$$
  $${with}_r\in {WITH}_R,{with}_x\in {WITH}_X,{with}_{and}\in {WITH}_{And}$$

  Composition of relations

  • Domain alignment in VAE latent space
    
    
    
  • Potential mapping
    
    
    

• Non-local modules
  
  
  
  
  Formula parameters:

  • F: feature map
  • m: Binary image with value 0-1

• Area defect detection
  Area defect detection is the automatic detection of the input guide mask m in the non-local module in the global branch.
  
  
  
  

• Progressive generator network for facial enhancement
  
  
  
  

• Synthetic data generation method
  

• result