ObjectAdd: Adding Objects into Image via a Training-Free Diffusion Modification Fashion

Ziyue Zhang,Mingbao Lin,Rongrong Ji

2024-05-02

Abstract:We introduce ObjectAdd, a training-free diffusion modification method to add user-expected objects into user-specified area. The motive of ObjectAdd stems from: first, describing everything in one prompt can be difficult, and second, users often need to add objects into the generated image. To accommodate with real world, our ObjectAdd maintains accurate image consistency after adding objects with technical innovations in: (1) embedding-level concatenation to ensure correct text embedding coalesce; (2) object-driven layout control with latent and attention injection to ensure objects accessing user-specified area; (3) prompted image inpainting in an attention refocusing & object expansion fashion to ensure rest of the image stays the same. With a text-prompted image, our ObjectAdd allows users to specify a box and an object, and achieves: (1) adding object inside the box area; (2) exact content outside the box area; (3) flawless fusion between the two areas

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper aims to address two main issues in text-to-image generation:

1. **Description Challenge**: Users find it difficult to accurately describe all details with a single text prompt, especially when it comes to expressing spatial relationships between objects in the image.
2. **Modification Limitation**: Users often need to make multiple modifications after generating an image to achieve the desired effect, but existing models have issues with consistency in the modified images.

To solve these problems, the paper proposes a method called **ObjectAdd**. This is a training-free diffusion model modification method that allows users to add desired objects to specified areas of an image while keeping the rest of the image unchanged. The key innovations of ObjectAdd include:

- **Text Embedding Cascade**: To address the embedding confusion caused by simple text concatenation, the paper proposes an embedding output cascade method to ensure that information from different texts is fused without interfering with each other.
- **Object-Driven Layout Control**: By injecting latent information and attention information, it ensures that the added objects are located in the user-specified area and naturally blend with the surrounding environment.
- **Prompt Image Repair**: Through attention refocusing and object expansion techniques, it maintains the consistency of the non-edited areas of the image.

The ObjectAdd method demonstrated good performance in experiments, particularly in maintaining the consistency of non-edited areas and high-quality addition of new objects. Compared with existing methods such as DALL-E 3, P2P, and Stable Diffusion v1-4, ObjectAdd showed superiority in multiple evaluation metrics.