FunEditor: Achieving Complex Image Edits via Function Aggregation with Diffusion Models

Mohammadreza Samadi,Fred X. Han,Mohammad Salameh,Hao Wu,Fengyu Sun,Chunhua Zhou,Di Niu

2024-12-18

Abstract:Diffusion models have demonstrated outstanding performance in generative tasks, making them ideal candidates for image editing. Recent studies highlight their ability to apply desired edits effectively by following textual instructions, yet with two key challenges remaining. First, these models struggle to apply multiple edits simultaneously, resulting in computational inefficiencies due to their reliance on sequential processing. Second, relying on textual prompts to determine the editing region can lead to unintended alterations to the image. We introduce FunEditor, an efficient diffusion model designed to learn atomic editing functions and perform complex edits by aggregating simpler functions. This approach enables complex editing tasks, such as object movement, by aggregating multiple functions and applying them simultaneously to specific areas. Our experiments demonstrate that FunEditor significantly outperforms recent inference-time optimization methods and fine-tuned models, either quantitatively across various metrics or through visual comparisons or both, on complex tasks like object movement and object pasting. In the meantime, with only 4 steps of inference, FunEditor achieves 5-24x inference speedups over existing popular methods. The code is available at: <a class="link-external link-http" href="http://mhmdsmdi.github.io/funeditor/" rel="external noopener nofollow">this http URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problems that this paper attempts to solve mainly focus on two aspects:

1. **Simultaneous application of multiple editing tasks**: Existing Diffusion Models (DM) have difficulty applying multiple editing operations simultaneously when performing image - editing tasks. These models usually rely on sequential processing methods, which lead to low computational efficiency and are prone to error accumulation.

2. **Region - editing problems based on text prompts**: Relying on text prompts to determine the editing region may lead to unexpected modifications to the image, because the text description may not be precise enough or may be ambiguous.

To solve these problems, the author proposes a new method named FunEditor. FunEditor can efficiently perform complex editing tasks by learning atomic editing functions and aggregating them. Specifically, it achieves the following goals:

- **Simultaneously applying multiple editing functions**: By combining multiple simple editing functions together and applying them simultaneously to a specific region, complex editing tasks (such as object movement) are achieved. This not only improves the editing efficiency but also reduces the error accumulation caused by sequential processing.

- **Improving editing precision and consistency**: Compared with methods relying on text prompts, FunEditor can more accurately control the editing range by directly specifying the editing region (using a binary mask), avoiding unnecessary impacts on non - target regions.

In addition, the experimental results show that when handling complex editing tasks (such as object movement and pasting), FunEditor not only significantly outperforms existing methods on multiple evaluation metrics but also can complete the editing in only 4 inference steps, which is 5 to 24 times faster than existing popular methods.

### Summary

The core problem of the paper is to improve the efficiency and accuracy of diffusion models in complex image - editing tasks, especially for the two key challenges of multi - task simultaneous editing and precise region editing. FunEditor successfully solves these problems by introducing a function - aggregation mechanism, demonstrating its superior performance in complex image - editing tasks.