Abstract:For image inpainting, the existing Denoising Diffusion Probabilistic Model (DDPM) based method i.e. RePaint can produce high-quality images for any inpainting form. It utilizes a pre-trained DDPM as a prior and generates inpainting results by conditioning on the reverse diffusion process, namely denoising process. However, this process is significantly time-consuming. In this paper, we propose an efficient DDPM-based image inpainting method which includes three speed-up strategies. First, we utilize a pre-trained Light-Weight Diffusion Model (LWDM) to reduce the number of parameters. Second, we introduce a skip-step sampling scheme of Denoising Diffusion Implicit Models (DDIM) for the denoising process. Finally, we propose Coarse-to-Fine Sampling (CFS), which speeds up inference by reducing image resolution in the coarse stage and decreasing denoising timesteps in the refinement stage. We conduct extensive experiments on both faces and general-purpose image inpainting tasks, and our method achieves competitive performance with approximately 60 times speedup.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to improve the speed of the image inpainting method based on the Denoising Diffusion Probabilistic Model (DDPM) while maintaining high - quality inpainting results. Specifically, although existing DDPM - based methods (such as RePaint) can generate high - quality image inpainting results, their inference processes are very time - consuming. For this reason, the author proposes an efficient DDPM - based image inpainting method, which significantly reduces the inference time through the following three acceleration strategies: 1. **Light - Weight Diffusion Model (LWDM)**: Accelerate the inference process by reducing the number of model parameters, and combine the modified training objective function to compensate for the performance degradation caused by the reduction of parameters. 2. **Skip - Step Sampling**: Introduce the skip - step sampling scheme of DDIM (Denoising Diffusion Implicit Models) in the denoising process to further accelerate the inference process. 3. **Coarse - to - Fine Sampling (CFS)**: Divide the denoising process into two stages, namely the coarse stage and the refinement stage. In the coarse stage, reduce the image resolution and the number of denoising steps; in the refinement stage, utilize the image prior generated in the coarse stage, reduce the number of denoising steps and increase the number of Conditional Resampling Modules (CRM) to enhance the image details. Through these strategies, this method achieves approximately 60 - fold acceleration while maintaining the inpainting quality comparable to existing methods. The experimental results show that this method performs well in both facial and general - purpose image inpainting tasks.

Minutes to Seconds: Speeded-up DDPM-based Image Inpainting with Coarse-to-Fine Sampling

Multi-Step Denoising Scheduled Sampling: Towards Alleviating Exposure Bias for Diffusion Models

TD-Paint: Faster Diffusion Inpainting Through Time Aware Pixel Conditioning

RePaint: Inpainting using Denoising Diffusion Probabilistic Models

GradPaint: Gradient-guided inpainting with diffusion models

Towards Coherent Image Inpainting Using Denoising Diffusion Implicit Models

Intelligent Painter: Picture Composition With Resampling Diffusion Model

A Diffusion Model with A FFT for Image Inpainting

DiffGANPaint: Fast Inpainting Using Denoising Diffusion GANs

A Progressive Image Inpainting Algorithm with a Mask Auto-update Branch

Free-Form Image Inpainting with Separable Gate Encoder-Decoder Network

Uni-paint: A Unified Framework for Multimodal Image Inpainting with Pretrained Diffusion Model

Progressive Inpainting Strategy with Partial Convolutions Generative Networks (PPCGN).

A Double Feature Fusion Network with Progressive Learning for Sharper Inpainting

Fast-DDPM: Fast Denoising Diffusion Probabilistic Models for Medical Image-to-Image Generation

Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

Fast Diffusion Probabilistic Model Sampling through the lens of Backward Error Analysis

Coherent and Multi-modality Image Inpainting via Latent Space Optimization

High-Resolution Image Inpainting with Iterative Confidence Feedback and Guided Upsampling

A Hybrid Inpainting Model Combining Diffusion and Enhanced Exemplar Methods

MMGInpainting: Multi-Modality Guided Image Inpainting Based On Diffusion Models