Abstract:Proper inference of semantics is necessary for realistic image inpainting. Most image inpainting methods use deep generative models, which require large image datasets to predict and generate content. However, predicting the missing regions and generating coherent content is difficult due to limited control. Existing approaches include image-guided or text-guided image inpainting, but none of them has taken both image and text as the guidance signals, as far as we know. To fill this gap, we propose a multi-modality guided (MMG) image inpainting approach based on the diffusion model. This MMGInpainting method uses both image and text as guidance for generating content within the target area for inpainting, effectively integrating the semantic information conveyed by the guiding image or text into the content of the inpainted region. To construct MMGInpainting, we start by enhancing the U-Net backbone with a customized Nonlinear Activation Free Network (NAFNet). This adapted NAFNet incorporates an Anchored Stripe Attention mechanism, which utilizes anchor points to effectively model global contextual dependencies. To regulate inpainting, we use a Semantic Fusion Encoder to guide the inverse process of the diffusion model. The process is iteratively executed to denoise and generate the desired inpainting result. Additionally, we explore how different modes of meaning interact and coordinate to offer users useful guidance for a more manageable inpainting procedure. Experimental results demonstrate that our approach produces faithful results adhering to the guiding information, while significantly improving computational efficiency. Github Repository: https://github.com/skipper-zc/MMGInpainting/

Symmetric Connected U-Net with Multi-Head Self Attention (MHSA) and WGAN for Image Inpainting

UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation

Face Image Inpainting Based on Generative Adversarial Network

A Method for Face Image Inpainting Based on Autoencoder and Generative Adversarial Network.

Image Inpainting Based on Interactive Separation Network and Progressive Reconstruction Algorithm

Multi-Scale Generative Adversarial Networks Image Inpainting Algorithm*

Image inpainting method based on AU-GAN

Image Inpainting Based on Contextual Coherent Attention GAN

MSE-Net: Generative Image Inpainting with Multi-Scale Encoder

Deep Generative Network for Image Inpainting with Gradient Semantics and Spatial-Smooth Attention

AGG: Attention-Based Gated Convolutional GAN with Prior Guidance for Image Inpainting

A Context-Based Multi-Scale Discriminant Model for Natural Image Inpainting

Multi-scale Generative Adversarial Network with Multi-Head External Attention for Image Inpainting

Semantic Image Inpainting with Multi-Stage Feature Reasoning Generative Adversarial Network

Generative Adversarial Network Using Multi-modal Guidance for Ultrasound Images Inpainting.

HSVConnect: HSV guided enhanced content generation network for image inpainting

MMGInpainting: Multi-Modality Guided Image Inpainting Based On Diffusion Models

Multi-scale Generative Adversarial Inpainting Network Based on Cross-Layer Attention Transfer Mechanism.

Hourglass Attention Network for Image Inpainting.

An Adaptive Iterative Inpainting Method with More Information Exploration

DNNAM: Image inpainting algorithm via deep neural networks and attention mechanism