ImageInpaint：图像修复

前言

传统方法

Year	Proceeding	Title	Tag
2000	SIGGRAPH 2000	Image Inpainting [pdf]	Diffusion-based
2001	TIP 2001	Filling-in by joint interpolation of vector fields and gray levels [pdf]	Diffusion-based
2001	CVPR 2001	Navier-stokes, fluid dynamics, and image and video inpainting [pdf]
2001	SIGGRAPH 2001	Image Quilting for Texture Synthesis and Transfer [pdf]
2001	SIGGRAPH 2001	Synthesizing Natural Textures [pdf]
2002	EJAM 2002	Digital inpainting based on the mumford–shah–euler image model [pdf]	Diffusion-based
2003	CVPR 2003	Object removal by exemplar-based inpainting [pdf]
2003	TIP 2003	Simultaneous structure and texture image inpainting [pdf]	Diffusion-based
2003	TIP 2003	Structure and Texture Filling-In of Missing Image Blocks in Wireless Transmission and Compression Applications [pdf]
2003	ICCV 2003	Learning How to Inpaint from Global Image Statistics [pdf]	Diffusion-based
2003	TOG 2003	Fragment-based image completion [pdf]	Patch-based
2004	TIP 2004	Region Filling and Object Removal by Exemplar-Based Image Inpainting [pdf]	Patch-based; Inpainting order
2004	TPAMI 2004	Space-Time Video Completion [pdf]
2005	SIGGRAPH 2005	Image Completion with Structure Propagation [pdf]	Patch-based
2006	ISCS 2006	Image Compression with Structure Aware Inpainting [pdf]
2007	TOG 2007	Scene completion using millions of photographs [pdf]
2007	CSVT 2007	Image Compression With Edge-Based Inpainting [pdf]	Diffusion-based
2008	CVPR 2008	Summarizing Visual Data Using Bidirectional Similarity [pdf]
2009	SIGGRAPH 2009	PatchMatch: a randomized correspondence algorithm for structural image editing [pdf]	Patch-based
2010	TIP 2010	Image inpainting by patch propagation using patch sparsity [pdf]	Patch-based
2011	FTCGV 2011	Structured learning and prediction in computer vision [pdf]
2011	ICIP 2011	Examplar-based inpainting based on local geometry [pdf]	Inpainting order
2012	TOG 2012	Combining inconsistent images using patch-based synthesis[pdf]	Patch-based
2014	TOG 2014	Image completion using Planar structure guidance [pdf]	Patch-based
2014	TVCG 2014	High-Quality Real-Time Video Inpainting with PixMix [pdf]	Video
2014	SIAM 2014	Video inpainting of complex scenes [pdf]	Video
2015	TIP 2015	Annihilating Filter-Based Low-Rank Hankel Matrix Approach for Image Inpainting [pdf]
2015	TIP 2015	Exemplar-Based Inpainting: Technical Review and New Heuristics for Better Geometric Reconstructions [pdf]
2016	TOG 2016	Temporally coherent completion of dynamic video [pdf]	Video

深度学习

Year	Proceeding	Title	Tag
2012	NIPS 2012	Image denoising and inpainting with deep neural networks [pdf]
2014	GCPR 2014	Mask-specific inpainting with deep neural networks [pdf]
2014	NIPS 2014	Deep Convolutional Neural Network for Image Deconvolution [pdf]
2015	NIPS 2015	Shepard Convolutional Neural Networks [pdf] [code]
2016	CVPR 2016	Context Encoders: Feature Learning by Inpainting [pdf] [code]
2016	SIGGRAPH 2016	High-resolution multi-scale neural texture synthesis [pdf]
2017	CVPR 2017	Semantic image inpainting with deep generative models [pdf] [code]
2017	CVPR 2017	High-resolution image inpainting using multi-scale neural patch synthesis [pdf] [code]
2017	CVPR 2017	Generative Face Completion [pdf] [code]
2017	SIGGRAPH 2017	Globally and Locally Consistent Image Completion [pdf] [code]
2018	CVPR 2018	Generative Image Inpainting with Contextual Attention [pdf] [code]
2018	CVPR 2018	Natural and Effective Obfuscation by Head Inpainting [pdf]
2018	CVPR 2018	Eye In-Painting With Exemplar Generative Adversarial Networks [pdf] [code]
2018	CVPR 2018	UV-GAN: Adversarial Facial UV Map Completion for Pose-invariant Face Recognition [pdf]
2018	CVPR 2018	Disentangling Structure and Aesthetics for Style-aware Image Completion [pdf]
2018	ECCV 2018	Image Inpainting for Irregular Holes Using Partial Convolutions [pdf] [code]
2018	ECCV 2018	Contextual-based Image Inpainting: Infer, Match, and Translate [pdf]
2018	ECCV 2018	Shift-Net: Image Inpainting via Deep Feature Rearrangement [pdf] [code]
2018	NIPS 2018	Image Inpainting via Generative Multi-column Convolutional Neural Networks [pdf] [code]
2018	TOG 2018	Faceshop: Deep sketch-based face image editing [pdf]
2018	ACM MM 2018	Structural inpainting [pdf]
2018	ACM MM 2018	Semantic Image Inpainting with Progressive Generative Networks [pdf] [code]
2018	BMVC 2018	SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting [pdf]
2018	BMVC 2018	MC-GAN: Multi-conditional Generative Adversarial Network for Image Synthesi [pdf] [code]
2018	ACCV 2018	Face Completion iwht Semantic Knowledge and Collaborative Adversarial Learning [pdf]
2018	ICASSP 2018	Edge-Aware Context Encoder for Image Inpainting [pdf]
2018	ICPR 2018	Deep Structured Energy-Based Image Inpainting [pdf] [code]
2018	AISTATS 2019	Probabilistic Semantic Inpainting with Pixel Constrained CNNs [pdf]
2019	ICRA 2019	Empty Cities: Image Inpainting for a Dynamic-Object-Invariant Space [pdf]
2019	AAAI 2019	Video Inpainting by Jointly Learning Temporal Structure and Spatial Details [pdf]	Video
2019	CVPR 2019	Pluralistic Image Completion [pdf] [code] [project]
2019	CVPR 2019	Deep Reinforcement Learning of Volume-guided Progressive View Inpainting for 3D Point Scene Completion from a Single Depth Image [pdf]
2019	CVPR 2019	Foreground-aware Image Inpainting [pdf]
2019	CVPR 2019	Privacy Protection in Street-View Panoramas using Depth and Multi-View Imagery [pdf]
2019	CVPR 2019	Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting [pdf] [code]
2019	CVPR 2019	Deep Flow-Guided Video Inpainting [pdf] [project]	Video
2019	CVPR 2019	Deep video inapinting [pdf]	Video
2019	CVPRW 2019	VORNet: Spatio-temporally Consistent Video Inpainting for Object Removal [pdf]	Video
2019	TNNLS 2019	PEPSI++: Fast and Lightweight Network for Image Inpainting [pdf]
2019	IJCAI 2019	MUSICAL: Multi-Scale Image Contextual Attention Learning for Inpainting [pdf]
2019	IJCAI 2019	Generative Image Inpainting with Submanifold Alignment [pdf]
2019	ACM MM 2019	Progressive Image Inpainting with Full-Resolution Residual Network [pdf] [code]
2019	ACM MM 2019	Deep Fusion Network for Image Completion [pdf] [code]
2019	ACM MM 2019	GAIN: Gradient Augmented Inpainting Network for Irregular Holes [pdf]
2019	ACM MM 2019	Single-shot Semantic Image Inpainting with Densely Connected Generative Networks [pdf]
2019	ICCVW 2019	EdgeConnect: Generative Image Inpainting with Adversarial Edge Learning [pdf] [code]
2019	ICCV 2019	Coherent Semantic Attention for Image Inpainting [pdf] [code]
2019	ICCV 2019	StructureFlow: Image Inpainting via Structure-aware Appearance Flow [pdf] [code]
2019	ICCV 2019	Progressive Reconstruction of Visual Structure for Image Inpainting [pdf] [code]
2019	ICCV 2019	Localization of Deep Inpainting Using High-Pass Fully Convolutional Network [pdf]
2019	ICCV 2019	Image Inpainting with Learnable Bidirectional Attention Maps [pdf] [code]
2019	ICCV 2019	Free-Form Image Inpainting with Gated Convolution [pdf] [project]
2019	ICCV 2019	FiNet: Compatible and Diverse Fashion Image Inpainting [pdf]	Fashion
2019	ICCV 2019	SC-FEGAN: Face Editing Generative Adversarial Network with User’s Sketch and Color [pdf] [code]	Face
2019	ICCV 2019	Human Motion Prediction via Spatio-Temporal Inpainting [pdf] [code]	Motion
2019	ICCV 2019	Copy-and-Paste Networks for Deep Video Inpainting [pdf] [code]	Video
2019	ICCV 2019	Onion-Peel Networks for Deep Video Completion [pdf] [code]	Video
2019	ICCV 2019	Free-form Video Inpainting with 3D Gated Convolution and Temporal PatchGAN [pdf] [code]	Video
2019	ICCV 2019	An Internal Learning Approach to Video Inpainting [pdf]	Video
2019	ICCV 2019	Vision-Infused Deep Audio Inpainting [pdf] [code]	Audio
2019	AAAI 2020	Region Normalization for Image Inpainting [pdf] [code]
2019	AAAI 2020	Learning to Incorporate Structure Knowledge for Image Inpainting [pdf] [code]
2020	CVPR 2020	Prior Guided GAN Based Semantic Inpainting [pdf]
2020	CVPR 2020	UCTGAN: Diverse Image Inpainting based on Unsupervised Cross-Space Translation [pdf]
2020	CVPR 2020	Recurrent Feature Reasoning for Image Inpainting [pdf] [code]
2020	CVPR 2020	Contextual Residual Aggregation for Ultra High-Resolution Image Inpainting [pdf] [code]
2020	CVPR 2020	3D Photography using Context-aware Layered Depth Inpainting [pdf] [code]
2020	CVPR 2020	Learning Oracle Attention for High-fidelity Face Completion [pdf]
2020	ECCV 2020	Rethinking Image Inpainting via a Mutual Encoder-Decoder with Feature Equalizations [pdf] [code]
2020	ECCV 2020	Short-Term and Long-Term Context Aggregation Network for Video Inpainting	Video
2020	ECCV 2020	Learning Object Placement by Inpainting for Compositional Data Augmentation
2020	ECCV 2020	Learning Joint Spatial-Temporal Transformations for Video Inpainting [pdf] [code]	Video
2020	ECCV 2020	High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling [pdf]
2020	ECCV 2020	DVI: Depth Guided Video Inpainting for Autonomous Driving [pdf] [code]	Video
2020	ECCV 2020	VCNet: A Robust Approach to Blind Image Inpainting [pdf]
2020	ECCV 2020	Guidance and Evaluation: Semantic-Aware Image Inpainting for Mixed Scenes [pdf]
2021	WACV 2021	Hyperrealistic Image Inpainting with Hypergraphs [pdf] [code]
2021	CVPR 2021	Generating Diverse Structure for Image Inpainting With Hierarchical VQ-VAE [pdf] [code]
2021	CVPR 2021	Image Inpainting with External-internal Learning and Monochromic Bottleneck [pdf] [code]
2021	CVPR 2021	PD-GAN: Probabilistic Diverse GAN for Image Inpainting [pdf] [code]
2021	CVPR 2021	Image Inpainting Guided by Coherence Priors of Semantics and Textures [pdf]
2021	CVPR 2021	FaceInpainter: High Fidelity Face Adaptation to Heterogeneous Domains [pdf]
2021	CVPR 2021	TransFill: Reference-guided Image Inpainting by Merging Multiple Color and Spatial Transformations [pdf] [code]
2021	CVPR 2021	Prior Based Human Completion [pdf]	Human completion
2021	IJCAI 2021	Context-Aware Image Inpainting with Learned Semantic Priors [pdf] [code]
2021	IJCAI 2021	Noise Doesn’t Lie: Towards Universal Detection of Deep Inpainting [pdf]	Inpainting detection
2021	TCSVT 2021	IID-Net: Image Inpainting Detection via Neural Architecture Search and Attention [pdf] [code]	Inpainting detection
2021	WWW 2021	Progressive Semantic Reasoning for Image Inpainting [pdf] [code]
2021	ICCV 2021	Occlusion-Aware Video Object Inpainting [pdf]	Video
2021	ICCV 2021	Internal Video Inpainting by Implicit Long-range Propagation [pdf] [code]	Video
2021	ICCV 2021	Distillation-Guided Image Inpainting [pdf]
2021	ICCV 2021	Frequency-Aware Spatiotemporal Transformers for Video Inpainting Detection [pdf]	Inpainting detection
2021	ICCV 2021	SLIDE: Single Image 3D Photography With Soft Layering and Depth-Aware Inpainting [pdf] [project]
2021	ICCV 2021	FuseFormer: Fusing Fine-Grained Information in Transformers for Video Inpainting [pdf] [code]	Video
2021	ICCV 2021	WaveFill: A Wavelet-Based Generation Network for Image Inpainting [pdf]
2021	ICCV 2021	CR-Fill: Generative Image Inpainting With Auxiliary Contextual Reconstruction [pdf] [code]
2021	ICCV 2021	Learning a Sketch Tensor Space for Image Inpainting of Man-Made Scenes [pdf] [project]
2021	ICCV 2021	Parallel Multi-Resolution Fusion Network for Image Inpainting [pdf]
2021	ICCV 2021	Flow-Guided Video Inpainting With Scene Templates [pdf]
2021	ICCV 2021	High-Fidelity Pluralistic Image Completion With Transformers [pdf] [project]
2021	ICCV 2021	Learning High-Fidelity Face Texture Completion Without Complete Face Texture [pdf]	Face
2021	WACV 2022	Resolution-robust Large Mask Inpainting with Fourier Convolutions [pdf] [code]
2022	CVPR 2022	Dual-path Image Inpainting with Auxiliary GAN Inversion
2022	CVPR 2022	MAT: Mask-Aware Transformer for Large Hole Image Inpainting [pdf] [code]
2022	CVPR 2022	Incremental Transformer Structure Enhanced Image Inpainting with Masking Positional Encoding [pdf] [code]
2022	CVPR 2022	Reduce Information Loss in Transformers for Pluralistic Image Inpainting
2022	CVPR 2022	MISF: Multi-level Interactive Siamese Filtering for High-Fidelity Image Inpainting [pdf] [code]
2022	CVPR 2022	RePaint: Inpainting using Denoising Diffusion Probabilistic Models [pdf] [code]
2022	CVPR 2022	DLFormer:Discrete Latent Transformer for Video Inpainting	Video
2022	CVPR 2022	The DEVIL is in the Details: A Diagnostic Evaluation Benchmark for Video Inpainting [pdf] [code]	Video
2022	CVPR 2022	Towards An End-to-End Framework for Flow-Guided Video Inpainting [pdf] [code]	Video
2022	CVPR 2022	Inertia-Guided Flow Completion and Style Fusion for Video Inpainting	Video
2022	CVPR 2022	DLFormer:Discrete Latent Transformer for Video Inpainting	Video

论文研读

今天这篇论文的主角是Image Inpainting via Generative Multi-column Convolutional Neural Networks，相关代码可以参考我的github：Code。

这篇论文提出了generative multi-column network（GMCNN）。摘要如下：

In this paper, we propose a generative multi-column network for image inpainting. This network synthesizes different image components in a parallel manner within one stage. To better characterize global structures, we design a confidence-driven reconstruction loss while an implicit diversified MRF regularization is adopted to enhance local details. The multi-column network combined with the reconstruction and MRF loss propagates local and global information derived from context to the target inpainting regions. Extensive experiments on challenging street view, face, natural objects and scenes manifest that our method produces visual compelling results even without previously common post-processing.

图像复原的概念：

Image inpainting (also known as image completion) aims to estimate suitable pixel information to fill holes in images. It serves various applications such as object removal, image restoration, image denoising, to name a few.

经典的图像复原方法主要定位以下三个重要的问题：

1. 提取合适的特征分析patch之间的相似性
2. 找到近邻的patches
3. 汇总辅助信息
   
   
   
   

方法

该网络主要有三部分：

1. a generator to produce results
2. global and local discriminators for adversarial training
3. a pretrained VGG network to calculate ID-MRF loss

网络结构

• generator
  

• ID-MRF Regularization
  马尔科夫随机场项
  
  

• Information Fusion
  
  

参考

1. Image Inpainting
2. F. Agostinelli, M. R. Anderson, and H. Lee. Adaptive multi-column deep neural networks with application to robust image denoising. In NIPS, pages 1493–1501, 2013
3. D. Ciregan, U. Meier, and J. Schmidhuber. Multi-column deep neural networks for image classification. In CVPR, pages 3642–3649. IEEE, 2012
4. Y. Zhang, D. Zhou, S. Chen, S. Gao, and Y. Ma. Single-image crowd counting via multi-column convolutional neural network. In CVPR, pages 589–597, 2016.