AnyControl: Create Your Artwork with Versatile Control on Text-to-Image Generation

Yanan Sun,Yanchen Liu,Yinhao Tang,Wenjie Pei,Kai Chen

2024-07-18

Abstract:The field of text-to-image (T2I) generation has made significant progress in recent years, largely driven by advancements in diffusion models. Linguistic control enables effective content creation, but struggles with fine-grained control over image generation. This challenge has been explored, to a great extent, by incorporating additional user-supplied spatial conditions, such as depth maps and edge maps, into pre-trained T2I models through extra encoding. However, multi-control image synthesis still faces several challenges. Specifically, current approaches are limited in handling free combinations of diverse input control signals, overlook the complex relationships among multiple spatial conditions, and often fail to maintain semantic alignment with provided textual prompts. This can lead to suboptimal user experiences. To address these challenges, we propose AnyControl, a multi-control image synthesis framework that supports arbitrary combinations of diverse control signals. AnyControl develops a novel Multi-Control Encoder that extracts a unified multi-modal embedding to guide the generation process. This approach enables a holistic understanding of user inputs, and produces high-quality, faithful results under versatile control signals, as demonstrated by extensive quantitative and qualitative evaluations. Our project page is available in <a class="link-external link-https" href="https://any-control.github.io" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address several key challenges in multi-control signal image synthesis:

1. **Input Flexibility**: Existing methods typically use fixed-length input channels or Mixture of Experts (MoE) designs when handling multiple control signals provided by users, which limits their adaptability to different input combinations. The number and modalities of control signals provided by users vary, which places high demands on the input flexibility of the model.

2. **Spatial Compatibility**: Control signals are not isolated; they collectively influence the overall layout of the generated image. Existing methods usually combine multiple conditions through weighted summation, which can easily lead to suboptimal fusion results and even cause some control signals to fail when dealing with occlusion relationships.

3. **Textual Compatibility**: Text descriptions usually determine the content of the generated image, while spatial conditions supplement structural information. However, the lack of effective communication between textual and spatial conditions often leads existing algorithms to prioritize spatial conditions when handling complex relationships, thereby neglecting the influence of textual prompts.

To address these challenges, the authors propose **AnyControl**, a controllable image synthesis framework that supports arbitrary combinations of multiple control signals. The core component of AnyControl is the **Multi-Control Encoder**, which extracts unified multi-modal embeddings by alternately using Multi-Control Fusion Blocks and Multi-Control Alignment Blocks, thereby achieving a comprehensive understanding of user inputs and generating high-quality, harmonious results.