Abstract:Image inpainting is an important image generation task, which aims to restore corrupted image from partial visible area. Recently, diffusion Schrödinger bridge methods effectively tackle this task by modeling the translation between corrupted and target images as a diffusion Schrödinger bridge process along a noising schedule path. Although these methods have shown superior performance, in this paper, we find that 1) existing methods suffer from a schedule-restoration mismatching issue, i.e., the theoretical schedule and practical restoration processes usually exist a large discrepancy, which theoretically results in the schedule not fully leveraged for restoring images; and 2) the key reason causing such issue is that the restoration process of all pixels are actually asynchronous but existing methods set a synchronous noise schedule to them, i.e., all pixels shares the same noise schedule. To this end, we propose a schedule-Asynchronous Diffusion Schrödinger Bridge (AsyncDSB) for image inpainting. Our insight is preferentially scheduling pixels with high frequency (i.e., large gradients) and then low frequency (i.e., small gradients). Based on this insight, given a corrupted image, we first train a network to predict its gradient map in corrupted area. Then, we regard the predicted image gradient as prior and design a simple yet effective pixel-asynchronous noise schedule strategy to enhance the diffusion Schrödinger bridge. Thanks to the asynchronous schedule at pixels, the temporal interdependence of restoration process between pixels can be fully characterized for high-quality image inpainting. Experiments on real-world datasets show that our AsyncDSB achieves superior performance, especially on FID with around 3% - 14% improvement over state-of-the-art baseline methods.

What problem does this paper attempt to address?

This paper attempts to solve a key problem in the image inpainting task, namely the mismatch between the existing methods in noise schedule and the actual inpainting process. Specifically: 1. **Problems of Existing Methods**: - Existing diffusion Schrödinger bridge methods (such as I2SB) usually use a symmetric noise schedule, which assumes that the inpainting process of all pixels is synchronous. - In fact, pixels of different frequencies (such as high - frequency edge structures and low - frequency color details) are asynchronous in the inpainting process. High - frequency pixels are usually repaired first, while low - frequency pixels are repaired later. - This mismatch leads to a large difference between the theoretical noise schedule and the actual inpainting process, thus limiting the quality of image inpainting. 2. **Solutions in the Paper**: - A new asynchronous noise schedule strategy (AsyncDSB) is proposed, which dynamically adjusts the noise schedule of each pixel according to the frequency information of pixels (such as gradient magnitude). - Specific steps include: - **Step 1: Image Gradient Completion**: First, predict the gradient map of the missing area through an adversarial generative network (GAN) as frequency prior information. - **Step 2: Gradient - Guided Asynchronous Diffusion Schrödinger Bridge**: According to the predicted gradient map, assign an asynchronous noise schedule to each pixel, so that high - frequency pixels are preferentially repaired and low - frequency pixels are repaired subsequently. 3. **Experimental Results**: - Extensive experiments were carried out on the CelebA - HQ and Places2 datasets. The results show that AsyncDSB has a significant improvement over the existing state - of - the - art methods in the FID metric, especially when dealing with different types of masks (such as central masks, half - masks, wide masks and narrow masks). - The quantitative results are shown in Table 1, and AsyncDSB has achieved the best or second - best results in multiple metrics. In conclusion, by introducing the asynchronous noise schedule strategy, this paper effectively solves the problem of mismatch between noise schedule and actual inpainting process in existing methods and significantly improves the quality of image inpainting.

AsyncDSB: Schedule-Asynchronous Diffusion Schrödinger Bridge for Image Inpainting

A Double Feature Fusion Network with Progressive Learning for Sharper Inpainting

Depth-Aided Inpainting for Disocclusion Restoration of Multi-View Images Using Depth-Image-Based Rendering

Implicit Image-to-Image Schrodinger Bridge for Image Restoration

Structure Matters: Tackling the Semantic Discrepancy in Diffusion Models for Image Inpainting

Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

A Novel DIBR Method Based on Image Inpainting

The Improved Image Inpainting Algorithm Via Encoder and Similarity Constraint

A Diffusion Model with A FFT for Image Inpainting

Partial Differential Equation Inpainting Method Based on Image Characteristics.

Diffusion Models with Anisotropic Gaussian Splatting for Image Inpainting

Implicit Image-to-Image Schrodinger Bridge for CT Super-Resolution and Denoising.

Anti-forensics of Diffusion-Based Image Inpainting.

Schedule Your Edit: A Simple yet Effective Diffusion Noise Schedule for Image Editing

Diverse Image Inpainting with Disentangled Uncertainty

Minutes to Seconds: Speeded-up DDPM-based Image Inpainting with Coarse-to-Fine Sampling

Image inpainting via progressive decoder and gradient guidance

Progressively Inpainting Images Based on a Forked-Then-Fused Decoder Network

Efficient Diffusion Model for Image Restoration by Residual Shifting

Art Image Inpainting with Style-guided Dual-branch Inpainting Network