Abstract:Comprehending natural language instructions is a charming property for both 2D and 3D layout synthesis systems. Existing methods implicitly model object joint distributions and express object relations, hindering generation's controllability. We introduce InstructLayout, a novel generative framework that integrates a semantic graph prior and a layout decoder to improve controllability and fidelity for 2D and 3D layout synthesis. The proposed semantic graph prior learns layout appearances and object distributions simultaneously, demonstrating versatility across various downstream tasks in a zero-shot manner. To facilitate the benchmarking for text-driven 2D and 3D scene synthesis, we respectively curate two high-quality datasets of layout-instruction pairs from public Internet resources with large language and multimodal models. Extensive experimental results reveal that the proposed method outperforms existing state-of-the-art approaches by a large margin in both 2D and 3D layout synthesis tasks. Thorough ablation studies confirm the efficacy of crucial design components.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to generate controllable and high - quality 2D and 3D layouts through natural language instructions. Specifically, existing methods are implicit when modeling the joint distribution of objects and expressing object relationships, which hinders the controllability of the generated results. In addition, existing methods have difficulty handling complex natural language instructions, resulting in a lack of precision and diversity in the generated layouts. To solve these problems, the authors propose **INSTRUCT LAYOUT**, a new generation framework that combines semantic graph prior and layout decoder to improve the controllability and fidelity of 2D and 3D layout synthesis. The main contributions of this framework include: 1. **Introducing semantic graph prior**: Learn the layout appearance and object distribution, thereby achieving zero - shot generation in various downstream tasks. 2. **Constructing high - quality datasets**: To promote the benchmarking of text - driven 2D and 3D scene synthesis, the authors have collated two high - quality datasets from public Internet resources, containing a large number of layout - instruction pairs. 3. **Improving the generation model**: Through extensive experimental verification, this method significantly outperforms the existing state - of - the - art methods in 2D and 3D layout synthesis tasks. ### Specific problem description 1. **Understanding natural language instructions**: Existing layout synthesis systems have difficulty accurately understanding natural language instructions, resulting in a lack of controllability in the generated layouts. 2. **Improving generation quality**: Existing methods cannot ensure high fidelity and controllability simultaneously when generating layouts. 3. **Modeling complex object relationships**: Existing methods have difficulty handling complex object relationships, resulting in a lack of diversity and precision in the generated layouts. ### Solution **INSTRUCT LAYOUT** solves the above problems in the following ways: 1. **Semantic graph prior**: Use semantic graph prior to model object attributes and layout distribution, thereby achieving zero - shot generation in various downstream tasks. 2. **Layout decoder**: Generate specific layout configurations through the decoder to ensure that the generated layouts conform to the semantic graph prior and are close to the instructions provided by the user. 3. **Two - stage scheme**: Process discrete and continuous attributes separately, reducing the burden of network optimization and improving the quality and controllability of generation. ### Summary The core problem of this paper is to generate controllable and high - quality 2D and 3D layouts through natural language instructions, and **INSTRUCT LAYOUT** successfully solves the deficiencies of existing methods in controllability and generation quality by introducing semantic graph prior and layout decoder.

InstructLayout: Instruction-Driven 2D and 3D Layout Synthesis with Semantic Graph Prior

InstructScene: Instruction-Driven 3D Indoor Scene Synthesis with Semantic Graph Prior

Learning 3 D Scene Synthesis from Annotated RGB-D Images

Layout-Bridging Text-to-Image Synthesis

Layout2Scene: 3D Semantic Layout Guided Scene Generation via Geometry and Appearance Diffusion Priors

LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation

Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models

Layout-your-3D: Controllable and Precise 3D Generation with 2D Blueprint

Learn and Sample Together: Collaborative Generation for Graphic Design Layout.

LayoutGPT: Compositional Visual Planning and Generation with Large Language Models

Planner3D: LLM-enhanced graph prior meets 3D indoor scene explicit regularization

Training-free Composite Scene Generation for Layout-to-Image Synthesis

Reason out Your Layout: Evoking the Layout Master from Large Language Models for Text-to-Image Synthesis

Explicitly Representing Syntax Improves Sentence-to-layout Prediction of Unexpected Situations

PosterLayout: A New Benchmark and Approach for Content-aware Visual-Textual Presentation Layout

CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation

Urban Architect: Steerable 3D Urban Scene Generation with Layout Prior

LayoutVLM: Differentiable Optimization of 3D Layout via Vision-Language Models

A Parse-Then-Place Approach for Generating Graphic Layouts from Textual Descriptions

Diagnostic Benchmark and Iterative Inpainting for Layout-Guided Image Generation

LayoutLLM: Layout Instruction Tuning with Large Language Models for Document Understanding