Abstract:Recent years have witnessed the success of large text-to-image diffusion models and their remarkable potential to generate high-quality images. The further pursuit of enhancing the editability of images has sparked significant interest in the downstream task of inpainting a novel object described by a text prompt within a designated region in the image. Nevertheless, the problem is not trivial from two aspects: 1) Solely relying on one single U-Net to align text prompt and visual object across all the denoising timesteps is insufficient to generate desired objects; 2) The controllability of object generation is not guaranteed in the intricate sampling space of diffusion model. In this paper, we propose to decompose the typical single-stage object inpainting into two cascaded processes: 1) semantic pre-inpainting that infers the semantic features of desired objects in a multi-modal feature space; 2) high-fieldity object generation in diffusion latent space that pivots on such inpainted semantic features. To achieve this, we cascade a Transformer-based semantic inpainter and an object inpainting diffusion model, leading to a novel CAscaded Transformer-Diffusion (CAT-Diffusion) framework for text-guided object inpainting. Technically, the semantic inpainter is trained to predict the semantic features of the target object conditioning on unmasked context and text prompt. The outputs of the semantic inpainter then act as the informative visual prompts to guide high-fieldity object generation through a reference adapter layer, leading to controllable object inpainting. Extensive evaluations on OpenImages-V6 and MSCOCO validate the superiority of CAT-Diffusion against the state-of-the-art methods. Code is available at \url{<a class="link-external link-https" href="https://github.com/Nnn-s/CATdiffusion" rel="external noopener nofollow">this https URL</a>}.

Mutual Dual-task Generator with Adaptive Attention Fusion for Image Inpainting

UCTGAN: Diverse Image Inpainting Based on Unsupervised Cross-Space Translation

MIGT: Multi-modal Image Inpainting Guided with Text.

MMGInpainting: Multi-Modality Guided Image Inpainting Based On Diffusion Models

Improving Text-guided Object Inpainting with Semantic Pre-inpainting

Dual-path Image Inpainting with Auxiliary GAN Inversion

Both Spatial and Frequency Cues Contribute to High-Fidelity Image Inpainting

DAFT-GAN: Dual Affine Transformation Generative Adversarial Network for Text-Guided Image Inpainting

Unbiased Multi-Modality Guidance for Image Inpainting

Generative Image Inpainting with Segmentation Confusion Adversarial Training and Contrastive Learning

Face Image Inpainting Based on Generative Adversarial Network

Boosted GAN with Semantically Interpretable Information for Image Inpainting

An Adaptive Iterative Inpainting Method with More Information Exploration

SPG-Net: Segmentation Prediction and Guidance Network for Image Inpainting

Guiding image inpainting via structure and texture features with dual encoder

Attentional coarse-and-fine generative adversarial networks for image inpainting

MFMAM: Image inpainting via multi-scale feature module with attention module

Semantic Image Inpainting with Multi-Stage Feature Reasoning Generative Adversarial Network

Dual Attention GANs for Semantic Image Synthesis

Delving Globally into Texture and Structure for Image Inpainting

Generative Image Inpainting with Residual Attention Learning