Good Seed Makes a Good Crop: Discovering Secret Seeds in Text-to-Image Diffusion Models

Katherine Xu,Lingzhi Zhang,Jianbo Shi

2024-05-24

Abstract:Recent advances in text-to-image (T2I) diffusion models have facilitated creative and photorealistic image synthesis. By varying the random seeds, we can generate various images for a fixed text prompt. Technically, the seed controls the initial noise and, in multi-step diffusion inference, the noise used for reparameterization at intermediate timesteps in the reverse diffusion process. However, the specific impact of the random seed on the generated images remains relatively unexplored. In this work, we conduct a large-scale scientific study into the impact of random seeds during diffusion inference. Remarkably, we reveal that the best 'golden' seed achieved an impressive FID of 21.60, compared to the worst 'inferior' seed's FID of 31.97. Additionally, a classifier can predict the seed number used to generate an image with over 99.9% accuracy in just a few epochs, establishing that seeds are highly distinguishable based on generated images. Encouraged by these findings, we examined the influence of seeds on interpretable visual dimensions. We find that certain seeds consistently produce grayscale images, prominent sky regions, or image borders. Seeds also affect image composition, including object location, size, and depth. Moreover, by leveraging these 'golden' seeds, we demonstrate improved image generation such as high-fidelity inference and diversified sampling. Our investigation extends to inpainting tasks, where we uncover some seeds that tend to insert unwanted text artifacts. Overall, our extensive analyses highlight the importance of selecting good seeds and offer practical utility for image generation.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

### What problem does this paper attempt to solve?

This paper primarily explores the impact of random seeds in Text-to-Image (T2I) diffusion models. Specifically:

1. **Impact of Random Seeds on Image Quality**:
- The study finds significant differences in the quality of images generated with different random seeds. For example, "golden seeds" can produce high-quality images with FID scores as low as 21.60, while "poor seeds" have FID scores as high as 31.97.

2. **Seed Distinguishability**:
- By training a classifier to predict the seed number used to generate an image, the results show that these seeds can be highly distinguished through the generated images. The classifier achieved a validation accuracy of over 99.9% after just 6 rounds of training.

3. **Impact of Seeds on Interpretable Visual Dimensions**:
- Seeds not only affect the overall quality of the image but also influence the style, layout, and whether text artifacts are produced. For instance, some seeds tend to generate grayscale images or images with prominent sky regions.

4. **Seed-Based Optimization Applications**:
- Utilizing these findings, the researchers propose several downstream applications, such as High-Fidelity Inference, which improves image quality by selecting "golden seeds," and Diversified Sampling, which enhances the diversity of image styles and layouts.

In summary, this paper reveals the significant role of random seeds in T2I diffusion models through large-scale experiments and proposes methods to improve image generation quality by leveraging these findings.