Abstract:Despite significant advancements in text-to-image models for generating high-quality images, these methods still struggle to ensure the controllability of text prompts over images in the context of complex text prompts, especially when it comes to retaining object attributes and relationships. In this paper, we propose CompAgent, a training-free approach for compositional text-to-image generation, with a large language model (LLM) agent as its core. The fundamental idea underlying CompAgent is premised on a divide-and-conquer methodology. Given a complex text prompt containing multiple concepts including objects, attributes, and relationships, the LLM agent initially decomposes it, which entails the extraction of individual objects, their associated attributes, and the prediction of a coherent scene layout. These individual objects can then be independently conquered. Subsequently, the agent performs reasoning by analyzing the text, plans and employs the tools to compose these isolated objects. The verification and human feedback mechanism is finally incorporated into our agent to further correct the potential attribute errors and refine the generated images. Guided by the LLM agent, we propose a tuning-free multi-concept customization model and a layout-to-image generation model as the tools for concept composition, and a local image editing method as the tool to interact with the agent for verification. The scene layout controls the image generation process among these tools to prevent confusion among multiple objects. Extensive experiments demonstrate the superiority of our approach for compositional text-to-image generation: CompAgent achieves more than 10\% improvement on T2I-CompBench, a comprehensive benchmark for open-world compositional T2I generation. The extension to various related tasks also illustrates the flexibility of our CompAgent for potential applications.

Improving Compositional Text-to-image Generation with Large Vision-Language Models

Progressive Compositionality In Text-to-Image Generative Models

ARTIST: Improving the Generation of Text-rich Images with Disentangled Diffusion Models and Large Language Models

IterComp: Iterative Composition-Aware Feedback Learning from Model Gallery for Text-to-Image Generation

CoVLM: Composing Visual Entities and Relationships in Large Language Models Via Communicative Decoding

ComAlign: Compositional Alignment in Vision-Language Models

Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis

RealignDiff: Boosting Text-to-Image Diffusion Model with Coarse-to-fine Semantic Re-alignment

Improving Visual Storytelling with Multimodal Large Language Models

InternLM-XComposer: A Vision-Language Large Model for Advanced Text-image Comprehension and Composition

Divide and Conquer: Language Models can Plan and Self-Correct for Compositional Text-to-Image Generation

RealCompo: Balancing Realism and Compositionality Improves Text-to-Image Diffusion Models

Paragraph-to-Image Generation with Information-Enriched Diffusion Model

FINECAPTION: Compositional Image Captioning Focusing on Wherever You Want at Any Granularity

Vision-Language Matching for Text-to-Image Synthesis via Generative Adversarial Networks

Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis

Prompting Large Vision-Language Models for Compositional Reasoning

MMCOMPOSITION: Revisiting the Compositionality of Pre-trained Vision-Language Models

Natural Language Inference Improves Compositionality in Vision-Language Models