Abstract:Introducing user-specified visual concepts in image editing is highly practical as these concepts convey the user's intent more precisely than text-based descriptions. We propose FreeEdit, a novel approach for achieving such reference-based image editing, which can accurately reproduce the visual concept from the reference image based on user-friendly language instructions. Our approach leverages the multi-modal instruction encoder to encode language instructions to guide the editing process. This implicit way of locating the editing area eliminates the need for manual editing masks. To enhance the reconstruction of reference details, we introduce the Decoupled Residual ReferAttention (DRRA) module. This module is designed to integrate fine-grained reference features extracted by a detail extractor into the image editing process in a residual way without interfering with the original self-attention. Given that existing datasets are unsuitable for reference-based image editing tasks, particularly due to the difficulty in constructing image triplets that include a reference image, we curate a high-quality dataset, FreeBench, using a newly developed twice-repainting scheme. FreeBench comprises the images before and after editing, detailed editing instructions, as well as a reference image that maintains the identity of the edited object, encompassing tasks such as object addition, replacement, and deletion. By conducting phased training on FreeBench followed by quality tuning, FreeEdit achieves high-quality zero-shot editing through convenient language instructions. We conduct extensive experiments to evaluate the effectiveness of FreeEdit across multiple task types, demonstrating its superiority over existing methods. The code will be available at: <a class="link-external link-https" href="https://freeedit.github.io/" rel="external noopener nofollow">this https URL</a>.

SpecRef: A Fast Training-free Baseline of Specific Reference-Condition Real Image Editing

FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction

FastEdit: Fast Text-Guided Single-Image Editing via Semantic-Aware Diffusion Fine-Tuning

Zero-shot Image Editing with Reference Imitation

Conditional Text-to-Image Generation with Reference Guidance

Unified Diffusion-Based Rigid and Non-Rigid Editing with Text and Image Guidance

ColorEdit: Training-free Image-Guided Color editing with diffusion model

EasyRef: Omni-Generalized Group Image Reference for Diffusion Models via Multimodal LLM

ReEdit: Multimodal Exemplar-Based Image Editing with Diffusion Models

RefDrop: Controllable Consistency in Image or Video Generation via Reference Feature Guidance

AdapEdit: Spatio-Temporal Guided Adaptive Editing Algorithm for Text-Based Continuity-Sensitive Image Editing

Real-time 3D-aware Portrait Editing from a Single Image

Where You Edit is What You Get: Text-guided Image Editing with Region-Based Attention.

Edicho: Consistent Image Editing in the Wild

Prompt Tuning Inversion for Text-Driven Image Editing Using Diffusion Models

Towards Real-time Text-driven Image Manipulation with Unconditional Diffusion Models

Prompt-to-prompt image editing with cross attention control

StyleDiffusion: Prompt-Embedding Inversion for Text-Based Editing

InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions

Tuning-Free Image Customization with Image and Text Guidance