Towards Better Text-to-Image Generation Alignment via Attention Modulation

Yihang Wu,Xiao Cao,Kaixin Li,Zitan Chen,Haonan Wang,Lei Meng,Zhiyong Huang

2024-04-22

Abstract:In text-to-image generation tasks, the advancements of diffusion models have facilitated the fidelity of generated results. However, these models encounter challenges when processing text prompts containing multiple entities and attributes. The uneven distribution of attention results in the issues of entity leakage and attribute misalignment. Training from scratch to address this issue requires numerous labeled data and is resource-consuming. Motivated by this, we propose an attribution-focusing mechanism, a training-free phase-wise mechanism by modulation of attention for diffusion model. One of our core ideas is to guide the model to concentrate on the corresponding syntactic components of the prompt at distinct timesteps. To achieve this, we incorporate a temperature control mechanism within the early phases of the self-attention modules to mitigate entity leakage issues. An object-focused masking scheme and a phase-wise dynamic weight control mechanism are integrated into the cross-attention modules, enabling the model to discern the affiliation of semantic information between entities more effectively. The experimental results in various alignment scenarios demonstrate that our model attain better image-text alignment with minimal additional computational cost.

Computation and Language,Artificial Intelligence,Multimedia

What problem does this paper attempt to address?

This paper attempts to solve the entity leakage and attribute misalignment problems that models encounter when dealing with text prompts containing multiple entities and attributes in text - to - image generation tasks. Specifically:

1. **Entity Leakage**: During the generation process, the attention distribution among different objects is confused, resulting in blurred entity boundaries or incorrect entity fusion.
2. **Attribute Misalignment**: The attention response spreads among different objects, leading to incorrect binding of object attributes.

These problems are particularly prominent in existing diffusion models, especially when dealing with complex multi - entity and multi - attribute text prompts. To solve these problems, the authors propose a training - free phased mechanism based on attention modulation, which mainly includes the following parts:

1. **Self - Attention Temperature Control**: By introducing a temperature control mechanism in the self - attention module, the attention distribution is adjusted to reduce the entity leakage problem.
2. **Object - Focusing Mask**: An object - focusing mask mechanism is introduced in the cross - attention module to ensure that each patch mainly focuses on a single entity group, reducing attribute misalignment.
3. **Phased Dynamic Weight Control**: In different stages of generation, the weights of different semantic components are dynamically adjusted to improve attribute alignment.

These methods aim to improve the consistency between the generated image and the text prompt while maintaining a low computational cost. Experimental results show that this method achieves better image - text alignment effects in multiple alignment scenarios.