Enhancing Compositional Text-to-Image Generation with Reliable Random Seeds

Shuangqi Li,Hieu Le,Jingyi Xu,Mathieu Salzmann

2024-11-28

Abstract:Text-to-image diffusion models have demonstrated remarkable capability in generating realistic images from arbitrary text prompts. However, they often produce inconsistent results for compositional prompts such as "two dogs" or "a penguin on the right of a bowl". Understanding these inconsistencies is crucial for reliable image generation. In this paper, we highlight the significant role of initial noise in these inconsistencies, where certain noise patterns are more reliable for compositional prompts than others. Our analyses reveal that different initial random seeds tend to guide the model to place objects in distinct image areas, potentially adhering to specific patterns of camera angles and image composition associated with the seed. To improve the model's compositional ability, we propose a method for mining these reliable cases, resulting in a curated training set of generated images without requiring any manual annotation. By fine-tuning text-to-image models on these generated images, we significantly enhance their compositional capabilities. For numerical composition, we observe relative increases of 29.3% and 19.5% for Stable Diffusion and PixArt-{\alpha}, respectively. Spatial composition sees even larger gains, with 60.7% for Stable Diffusion and 21.1% for PixArt-{\alpha}.

Computer Vision and Pattern Recognition,Machine Learning

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: Existing text - to - image diffusion models lack consistency and accuracy in the generated results when dealing with compositional prompts (such as "two dogs" or "the penguin on the right of the bowl"). Specifically, these models are prone to errors when handling quantity and spatial relationship descriptions. Through research, the author found that the initial noise (i.e., the random seed) has a significant impact on the results generated by compositional prompts. Different initial random seeds will cause the model to place objects in different image areas, thereby affecting the accuracy of the generated image.

### Core problems of the paper

1. **Inconsistency of compositional prompts**
- Text - to - image generation models often produce inconsistent results when dealing with compositional prompts, such as the quantity of multiple objects or specific spatial relationships.
- For example, prompts such as "four unicorns" or "the pigeon on top of the basketball" may lead to generated images that do not meet expectations.

2. **Impact of initial noise**
- The initial noise (random seed) has an important impact on the quality and consistency of the generated image. Some noise patterns are more reliable than others and can better generate images that conform to compositional prompts.
- Different initial random seeds will guide the model to place objects in different image areas, forming a specific layout pattern, thus affecting the accuracy of the generated results.

### Solutions

To solve these problems, the author proposed a method to mine reliable initial random seeds and use these seeds to generate a high - quality training data set to improve the compositional generation ability of text - to - image models. The specific steps include:

1. **Mining reliable random seeds**
- By analyzing the impact of different random seeds on the generated image, screen out those seeds that can generate more accurate compositional images.

2. **Generating high - quality training data**
- Use these reliable random seeds to generate images and use them to fine - tune existing text - to - image models.

3. **Fine - tuning the model**
- Fine - tune the model on these automatically generated high - quality data to enhance its compositional generation ability.

### Experimental results

The experimental results show that sampling with reliable random seeds can significantly improve the accuracy of the generated image. For numerical combination tasks, the relative improvements of Stable Diffusion and PixArt - α are 29.3% and 19.5% respectively; for spatial combination tasks, the improvements are more obvious, being 60.7% and 21.1% respectively.

Through this method, the author not only improves the generation quality of the model under compositional prompts but also provides an effective way to generate high - quality training data without manual annotation.