Text-Driven Image Editing via Learnable Regions

Yuanze Lin,Yi-Wen Chen,Yi-Hsuan Tsai,Lu Jiang,Ming-Hsuan Yang

2024-04-03

Abstract:Language has emerged as a natural interface for image editing. In this paper, we introduce a method for region-based image editing driven by textual prompts, without the need for user-provided masks or sketches. Specifically, our approach leverages an existing pre-trained text-to-image model and introduces a bounding box generator to identify the editing regions that are aligned with the textual prompts. We show that this simple approach enables flexible editing that is compatible with current image generation models, and is able to handle complex prompts featuring multiple objects, complex sentences, or lengthy paragraphs. We conduct an extensive user study to compare our method against state-of-the-art methods. The experiments demonstrate the competitive performance of our method in manipulating images with high fidelity and realism that correspond to the provided language descriptions. Our project webpage can be found at:

Computer Vision and Pattern Recognition,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The problem addressed in this paper is text-driven image editing without the need for user-specified editing regions. Existing methods either require manual creation of masks (time-consuming) or rely on pixel-level masks for accuracy. The paper proposes a text-based image editing method utilizing learnable regions, which identifies editing regions aligned with text prompts through a pre-trained text-to-image model and a bounding box generator, allowing for local image editing while preserving image context. This method is applicable for handling complex prompts involving multiple objects, complex sentences, or long paragraphs, and has shown performance comparable to the current state-of-the-art methods in user studies.