Generative Active Learning for Image Synthesis Personalization

Xulu Zhang,Wengyu Zhang,Xiao-Yong Wei,Jinlin Wu,Zhaoxiang Zhang,Zhen Lei,Qing Li

2024-04-16

Abstract:This paper presents a pilot study that explores the application of active learning, traditionally studied in the context of discriminative models, to generative models. We specifically focus on image synthesis personalization tasks. The primary challenge in conducting active learning on generative models lies in the open-ended nature of querying, which differs from the closed form of querying in discriminative models that typically target a single concept. We introduce the concept of anchor directions to transform the querying process into a semi-open problem. We propose a direction-based uncertainty sampling strategy to enable generative active learning and tackle the exploitation-exploration dilemma. Extensive experiments are conducted to validate the effectiveness of our approach, demonstrating that an open-source model can achieve superior performance compared to closed-source models developed by large companies, such as Google's StyleDrop. The source code is available at

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper aims to explore the application of Active Learning methods in generative models, specifically focusing on the task of Image Synthesis Personalization (ISP). The main problems addressed include:

1. **Active Learning in Generative Models**: Traditional active learning methods are primarily used for discriminative models, with relatively fewer applications in generative models. This paper proposes a new direction-based uncertainty sampling strategy to adapt to the characteristics of generative models.

2. **Sample Selection Problem**: With a limited number of reference images, how to select the most informative samples from the newly generated samples for training to improve model performance. The paper defines query directions semi-openly through Anchor Directions and combines random noise to achieve sample diversity.

3. **Exploration and Exploitation Balance Problem**: During the iterative process, how to balance fidelity to existing reference images with the exploration of new directions. The paper proposes a balancing scheme by evaluating the importance of different iterations to weigh the contributions of reference images and new directions.

Through these methods, the paper validates the effectiveness of its approach and demonstrates the potential of applying active learning in generative models.