Abstract:Proper inference of semantics is necessary for realistic image inpainting. Most image inpainting methods use deep generative models, which require large image datasets to predict and generate content. However, predicting the missing regions and generating coherent content is difficult due to limited control. Existing approaches include image-guided or text-guided image inpainting, but none of them has taken both image and text as the guidance signals, as far as we know. To fill this gap, we propose a multi-modality guided (MMG) image inpainting approach based on the diffusion model. This MMGInpainting method uses both image and text as guidance for generating content within the target area for inpainting, effectively integrating the semantic information conveyed by the guiding image or text into the content of the inpainted region. To construct MMGInpainting, we start by enhancing the U-Net backbone with a customized Nonlinear Activation Free Network (NAFNet). This adapted NAFNet incorporates an Anchored Stripe Attention mechanism, which utilizes anchor points to effectively model global contextual dependencies. To regulate inpainting, we use a Semantic Fusion Encoder to guide the inverse process of the diffusion model. The process is iteratively executed to denoise and generate the desired inpainting result. Additionally, we explore how different modes of meaning interact and coordinate to offer users useful guidance for a more manageable inpainting procedure. Experimental results demonstrate that our approach produces faithful results adhering to the guiding information, while significantly improving computational efficiency. Github Repository: https://github.com/skipper-zc/MMGInpainting/

Deep Generative Network for Image Inpainting with Gradient Semantics and Spatial-Smooth Attention

Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

Face Image Inpainting Based on Generative Adversarial Network

Improved Semantic Image Inpainting Method with Deep Convolution Generative Adversarial Networks.

ESGAN: Edge Loss and Spatial Convolution Generative Adversarial Network for Image Inpainting

DeepGIN: Deep Generative Inpainting Network for Extreme Image Inpainting

Multi-scale Gated Inpainting Network with Patch-Wise Spacial Attention.

Generative Image Inpainting with Residual Attention Learning

Deep Image Inpainting with Enhanced Normalization and Contextual Attention

SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

Deep Inception Generative Network for Cognitive Image Inpainting

Context-Aware Semantic Inpainting

Single-shot Semantic Image Inpainting with Densely Connected Generative Networks

AGG: Attention-Based Gated Convolutional GAN with Prior Guidance for Image Inpainting

Image Fine-grained Inpainting.

Interactive Image Inpainting Using Semantic Guidance

Image Inpainting Using Wasserstein Generative Adversarial Network

Image Inpainting Based on Double Joint Predictive Filtering and Wasserstein Generative Adversarial Networks.

Attentional coarse-and-fine generative adversarial networks for image inpainting

Boosted GAN with Semantically Interpretable Information for Image Inpainting

MMGInpainting: Multi-Modality Guided Image Inpainting Based On Diffusion Models