Abstract:Diffusion models have opened up new avenues for the field of image generation, resulting in the proliferation of high-quality models shared on open-source platforms. However, a major challenge persists in current text-to-image systems are often unable to handle diverse inputs, or are limited to single model results. Current unified attempts often fall into two orthogonal aspects: i) parse Diverse Prompts in input stage; ii) activate expert model to output. To combine the best of both worlds, we propose DiffusionGPT, which leverages Large Language Models (LLM) to offer a unified generation system capable of seamlessly accommodating various types of prompts and integrating domain-expert models. DiffusionGPT constructs domain-specific Trees for various generative models based on prior knowledge. When provided with an input, the LLM parses the prompt and employs the Trees-of-Thought to guide the selection of an appropriate model, thereby relaxing input constraints and ensuring exceptional performance across diverse domains. Moreover, we introduce Advantage Databases, where the Tree-of-Thought is enriched with human feedback, aligning the model selection process with human preferences. Through extensive experiments and comparisons, we demonstrate the effectiveness of DiffusionGPT, showcasing its potential for pushing the boundaries of image synthesis in diverse domains.

What problem does this paper attempt to address?

This paper attempts to solve two main problems in current text - to - image generation systems: 1. **Model Limitations**: Existing stable diffusion models (such as SD1.5) can adapt to a wide range of prompts, but perform poorly in specific domains. Conversely, domain - specific models (such as SD1.5+Lora) have excellent generation performance in specific sub - domains, but lack generality. 2. **Prompt Constraints**: When training stable diffusion models, text information usually consists of descriptive statements, such as titles. However, when using these models, various types of prompts will be encountered, including instructions and inspirations. Current generation models have difficulty achieving the best generation results when dealing with these diverse prompt types. To address these problems, the paper proposes a unified generation framework, **DiffusionGPT**, which uses large - language models (LLMs) to drive the entire text - to - image generation system. Specifically, the main contributions of DiffusionGPT include: - **New Insights**: DiffusionGPT uses large - language models (LLMs) as cognitive engines to process diverse inputs and select appropriate expert models for output. - **Integrated System**: DiffusionGPT provides a multi - functional and professional solution that is compatible with multiple diffusion models. Unlike existing methods that are limited to descriptive prompts, this framework can handle various types of prompts, expanding its application range. - **Efficient and Innovative**: DiffusionGPT has the characteristic of being training - free and can be easily integrated into plugins. By introducing Tree - of - Thought (ToT) and human feedback, the system achieves higher accuracy in the model selection process and pioneers a method for flexibly integrating more experts. - **High Effectiveness**: DiffusionGPT surpasses traditional stable diffusion models in multiple aspects, showing significant progress. By providing an integrated system, it provides a more efficient and effective path for the community development in the field of image generation. In conclusion, DiffusionGPT aims to solve the challenges in model limitations and prompt constraints in current text - to - image generation systems by combining large - language models and tree - like thinking structures, thereby improving the quality and diversity of generated images.

DiffusionGPT: LLM-Driven Text-to-Image Generation System

Emage: Non-Autoregressive Text-to-Image Generation

LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models

GlyphDiffusion: Text Generation as Image Generation

Mastering Text-to-Image Diffusion: Recaptioning, Planning, and Generating with Multimodal LLMs

A Survey of Diffusion Based Image Generation Models: Issues and Their Solutions

LLM4GEN: Leveraging Semantic Representation of LLMs for Text-to-Image Generation

ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models

Unified Multi-Modal Latent Diffusion for Joint Subject and Text Conditional Image Generation

Controllable Text-to-Image Generation with GPT-4

eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoisers

Guiding Text-to-Image Diffusion Model Towards Grounded Generation

Are Diffusion Models Vision-And-Language Reasoners?

Diffusion-Geo: A Two-Stage Controllable Text-To-Image Generative Model for Remote Sensing Scenarios

Implementing and Experimenting with Diffusion Models for Text-to-Image Generation

Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion

Decoder-Only LLMs Are Better Controllers for Diffusion Models

Diffusion-Adapter: Text Guided Image Manipulation with Frozen Diffusion Models

Dual Diffusion for Unified Image Generation and Understanding

Versatile Diffusion: Text, Images and Variations All in One Diffusion Model