StyleBooth: Image Style Editing with Multimodal Instruction

Zhen Han,Chaojie Mao,Zeyinzi Jiang,Yulin Pan,Jingfeng Zhang

2024-04-18

Abstract:Given an original image, image editing aims to generate an image that align with the provided instruction. The challenges are to accept multimodal inputs as instructions and a scarcity of high-quality training data, including crucial triplets of source/target image pairs and multimodal (text and image) instructions. In this paper, we focus on image style editing and present StyleBooth, a method that proposes a comprehensive framework for image editing and a feasible strategy for building a high-quality style editing dataset. We integrate encoded textual instruction and image exemplar as a unified condition for diffusion model, enabling the editing of original image following multimodal instructions. Furthermore, by iterative style-destyle tuning and editing and usability filtering, the StyleBooth dataset provides content-consistent stylized/plain image pairs in various categories of styles. To show the flexibility of StyleBooth, we conduct experiments on diverse tasks, such as text-based style editing, exemplar-based style editing and compositional style editing. The results demonstrate that the quality and variety of training data significantly enhance the ability to preserve content and improve the overall quality of generated images in editing tasks. Project page can be found at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily addresses two key issues in image style editing:

1. **Multimodal Input**: Most current methods only support single modality (text or image) instructions and cannot handle mixed instructions of text and image modalities simultaneously.
2. **Lack of High-Quality Training Data**: In general image editing scenarios, there is a shortage of high-quality images and their crucial paired data with multimodal (text and image) instructions.

To tackle these challenges, the paper proposes a unified approach named **StyleBooth**, which can support style editing with both text and image exemplars simultaneously, and designs a high-quality style editing dataset. Specifically:

- **Unified Conditional Framework**: Encodes text instructions and image exemplars into a unified condition for the diffusion model.
- **High-Quality Dataset Construction**: Constructs high-quality content-consistent stylized/non-stylized image pairs through iterative style de-stylization adjustments and usability filtering.
- **Multi-Task Experiments**: Demonstrates the capability in various tasks such as text-based style editing, image exemplar-based style editing, and combined style editing.

Experimental results show that high-quality and diverse training data significantly enhance the quality of generated images and the ability to retain content.