Abstract:Introducing user-specified visual concepts in image editing is highly practical as these concepts convey the user's intent more precisely than text-based descriptions. We propose FreeEdit, a novel approach for achieving such reference-based image editing, which can accurately reproduce the visual concept from the reference image based on user-friendly language instructions. Our approach leverages the multi-modal instruction encoder to encode language instructions to guide the editing process. This implicit way of locating the editing area eliminates the need for manual editing masks. To enhance the reconstruction of reference details, we introduce the Decoupled Residual ReferAttention (DRRA) module. This module is designed to integrate fine-grained reference features extracted by a detail extractor into the image editing process in a residual way without interfering with the original self-attention. Given that existing datasets are unsuitable for reference-based image editing tasks, particularly due to the difficulty in constructing image triplets that include a reference image, we curate a high-quality dataset, FreeBench, using a newly developed twice-repainting scheme. FreeBench comprises the images before and after editing, detailed editing instructions, as well as a reference image that maintains the identity of the edited object, encompassing tasks such as object addition, replacement, and deletion. By conducting phased training on FreeBench followed by quality tuning, FreeEdit achieves high-quality zero-shot editing through convenient language instructions. We conduct extensive experiments to evaluate the effectiveness of FreeEdit across multiple task types, demonstrating its superiority over existing methods. The code will be available at: <a class="link-external link-https" href="https://freeedit.github.io/" rel="external noopener nofollow">this https URL</a>.

Text-Guided Mask-free Local Image Retouching

Text-Guided Human Image Manipulation Via Image-Text Shared Space

MaTe3D: Mask-guided Text-based 3D-aware Portrait Editing

Mask-guided GAN for robust text editing in the scene

Zero-shot Text-driven Physically Interpretable Face Editing

FreeEdit: Mask-free Reference-based Image Editing with Multi-modal Instruction

Text-Driven Image Editing via Learnable Regions

FocalClick: Towards Practical Interactive Image Segmentation.

MAG-Edit: Localized Image Editing in Complex Scenarios via Mask-Based Attention-Adjusted Guidance

InstructEdit: Improving Automatic Masks for Diffusion-based Image Editing With User Instructions

TextMaster: Universal Controllable Text Edit

ChatFace: Chat-Guided Real Face Editing via Diffusion Latent Space Manipulation

Progressive Mask Transformer With Edge Enhancement for Image Manipulation Localization

Spatial Attention Guided Local Facial Attribute Editing

SAMControl: Controlling Pose and Object for Image Editing with Soft Attention Mask

Tuning-Free Image Customization with Image and Text Guidance

RetouchFormer: Semi-supervised High-Quality Face Retouching Transformer with Prior-Based Selective Self-Attention

Click2Mask: Local Editing with Dynamic Mask Generation

DesignEdit: Multi-Layered Latent Decomposition and Fusion for Unified & Accurate Image Editing

Where You Edit is What You Get: Text-guided Image Editing with Region-Based Attention.

HQRetouch: Learning Professional Face Retouching Via Masked Feature Fusion and Semantic-Aware Modulation