Abstract:Free-form image inpainting is the task of reconstructing parts of an image specified by an arbitrary binary mask. In this task, it is typically desired to generalize model capabilities to unseen mask types, rather than learning certain mask distributions. Capitalizing on the advances in diffusion models, in this paper, we propose a Denoising Diffusion Probabilistic Model (DDPM) based model capable of filling missing pixels fast as it models the backward diffusion process using the generator of a generative adversarial network (GAN) network to reduce sampling cost in diffusion models. Experiments on general-purpose image inpainting datasets verify that our approach performs superior or on par with most contemporary works.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is free - form image inpainting, that is, reconstructing the missing parts in an image according to a binary mask of an arbitrary shape. Specifically, the paper focuses on how to enable the model to generalize to unseen mask types, rather than just learning a specific mask distribution. To solve this problem, the authors propose a method based on the denoising diffusion probability model (DDPM) - DiffGANPaint, which utilizes the generator of the generative adversarial network (GAN) to accelerate the sampling process in the diffusion model, thereby achieving fast and high - quality image inpainting. ### Main Contributions 1. **Combining the Advantages of DDPM and GAN**: By using the trained DDPM for denoising and introducing the GAN generator in the reverse diffusion process, DiffGANPaint can improve the quality and speed of generated samples while maintaining structural consistency. 2. **Generalization Ability**: This method has good generalization ability for masks of arbitrary shapes and can handle unseen mask types during testing. 3. **Experimental Verification**: Through experiments on the general - purpose image inpainting dataset and the CelebA - HQ face dataset, it is proved that DiffGANPaint is superior to or comparable to most current methods in terms of visual quality and computational efficiency. ### Method Overview - **Denoising Diffusion Process**: First, use DDPM to denoise the input image to prepare the image for inpainting. - **Extract Masked Region**: Extract the region covered by the mask from the original image. - **GAN Generator Inpainting**: Use the GAN generator to inpaint the extracted masked region and finally generate a complete image. ### Experimental Results - **Visual Quality**: DiffGANPaint shows high visual quality on multiple datasets, especially in the case of limited computational resources. - **Computational Efficiency**: Compared with traditional diffusion models, DiffGANPaint can generate samples in fewer steps, although this may sacrifice some sample quality slightly. ### Conclusion The paper proposes an image inpainting method DiffGANPaint that combines DDPM and GAN. This method not only performs excellently in visual quality but also has a significant improvement in computational efficiency, and is suitable for multiple types of masks and datasets.

DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

Face Image Inpainting Based on Generative Adversarial Network

GradPaint: Gradient-guided inpainting with diffusion models

A Progressive Image Inpainting Algorithm with a Mask Auto-update Branch

Free-Form Image Inpainting with Gated Convolution

A Method for Face Image Inpainting Based on Autoencoder and Generative Adversarial Network.

Image Fine-grained Inpainting.

Progressive Inpainting Strategy with Partial Convolutions Generative Networks (PPCGN).

Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

Minutes to Seconds: Speeded-up DDPM-based Image Inpainting with Coarse-to-Fine Sampling

TD-Paint: Faster Diffusion Inpainting Through Time Aware Pixel Conditioning

Inst-Inpaint: Instructing to Remove Objects with Diffusion Models

Multi-scale Generative Adversarial Network with Multi-Head External Attention for Image Inpainting

Latent Denoising Diffusion GAN: Faster sampling, Higher image quality

MGAN-CRCM: A Novel Multiple Generative Adversarial Network and Coarse-Refinement Based Cognizant Method for Image Inpainting

DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting

Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models

Photo-Realistic and Robust Inpainting of Faces Using Refinement GANs

Contextual Attention Mechanism, SRGAN Based Inpainting System for Eliminating Interruptions from Images

A Diffusion Model with A FFT for Image Inpainting