Diffree: Text-Guided Shape Free Object Inpainting with Diffusion Model

Lirui Zhao,Tianshuo Yang,Wenqi Shao,Yuxin Zhang,Yu Qiao,Ping Luo,Kaipeng Zhang,Rongrong Ji

2024-07-24

Abstract:This paper addresses an important problem of object addition for images with only text guidance. It is challenging because the new object must be integrated seamlessly into the image with consistent visual context, such as lighting, texture, and spatial location. While existing text-guided image inpainting methods can add objects, they either fail to preserve the background consistency or involve cumbersome human intervention in specifying bounding boxes or user-scribbled masks. To tackle this challenge, we introduce Diffree, a Text-to-Image (T2I) model that facilitates text-guided object addition with only text control. To this end, we curate OABench, an exquisite synthetic dataset by removing objects with advanced image inpainting techniques. OABench comprises 74K real-world tuples of an original image, an inpainted image with the object removed, an object mask, and object descriptions. Trained on OABench using the Stable Diffusion model with an additional mask prediction module, Diffree uniquely predicts the position of the new object and achieves object addition with guidance from only text. Extensive experiments demonstrate that Diffree excels in adding new objects with a high success rate while maintaining background consistency, spatial appropriateness, and object relevance and quality.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper primarily focuses on addressing the problem of text-guided object addition in images. This task is challenging because it requires the new object to seamlessly integrate into the image while maintaining consistency in visual context such as lighting, texture, and spatial position.

To tackle this problem, the authors propose Diffree, a method that combines a diffusion model with an additional object mask prediction module, enabling shape-free object addition controlled solely by text. To train this model, the authors constructed a synthetic dataset named OABench, which contains 74K real-world samples. Each sample includes the original image, the image with an object removed, the mask of the object, and the object description.

Specifically, the main contributions of Diffree are as follows:

1. A model that enables text-guided shape-free object addition, allowing users to add objects without manually drawing masks.
2. The construction of a high-quality synthetic dataset, OABench, for training models on the object addition task.
3. The design of a set of evaluation rules to comprehensively assess the performance of different methods on the object addition task.

Experimental results show that Diffree significantly outperforms existing mask-based or text-based methods in terms of object addition success rate and performs excellently on multiple quantitative metrics.