LLM Blueprint: Enabling Text-to-Image Generation with Complex and Detailed Prompts

Hanan Gani,Shariq Farooq Bhat,Muzammal Naseer,Salman Khan,Peter Wonka

2024-02-26

Abstract:Diffusion-based generative models have significantly advanced text-to-image generation but encounter challenges when processing lengthy and intricate text prompts describing complex scenes with multiple objects. While excelling in generating images from short, single-object descriptions, these models often struggle to faithfully capture all the nuanced details within longer and more elaborate textual inputs. In response, we present a novel approach leveraging Large Language Models (LLMs) to extract critical components from text prompts, including bounding box coordinates for foreground objects, detailed textual descriptions for individual objects, and a succinct background context. These components form the foundation of our layout-to-image generation model, which operates in two phases. The initial Global Scene Generation utilizes object layouts and background context to create an initial scene but often falls short in faithfully representing object characteristics as specified in the prompts. To address this limitation, we introduce an Iterative Refinement Scheme that iteratively evaluates and refines box-level content to align them with their textual descriptions, recomposing objects as needed to ensure consistency. Our evaluation on complex prompts featuring multiple objects demonstrates a substantial improvement in recall compared to baseline diffusion models. This is further validated by a user study, underscoring the efficacy of our approach in generating coherent and detailed scenes from intricate textual inputs.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The paper attempts to address the challenges faced by current text-to-image generation techniques based on diffusion models when dealing with long and detailed text prompts. Specifically, while existing models excel at generating images from short, single-object descriptions, they often fail to faithfully capture all details when handling long text prompts that describe complex scenes with multiple objects. This can result in missing objects, inaccurate positioning, or generated objects that do not match the text description. To solve these issues, the paper proposes a new method that leverages large language models (LLMs) to extract key components from long text prompts, including the bounding box coordinates of foreground objects, detailed text descriptions of each object, and concise background context. These components form the foundation of a layout-to-image generation model, which operates in two stages: global scene generation and iterative refinement. Through this approach, the paper aims to improve the ability to generate images from long text prompts, ensuring that the final images faithfully reflect the complex text descriptions.