Abstract:Diffusion models have been recognized for their ability to generate images that are not only visually appealing but also of high artistic quality. As a result, Layout-to-Image (L2I) generation has been proposed to leverage region-specific positions and descriptions to enable more precise and controllable generation. However, previous methods primarily focus on UNet-based models (e.g., SD1.5 and SDXL), and limited effort has explored Multimodal Diffusion Transformers (MM-DiTs), which have demonstrated powerful image generation capabilities. Enabling MM-DiT for layout-to-image generation seems straightforward but is challenging due to the complexity of how layout is introduced, integrated, and balanced among multiple modalities. To this end, we explore various network variants to efficiently incorporate layout guidance into MM-DiT, and ultimately present SiamLayout. To Inherit the advantages of MM-DiT, we use a separate set of network weights to process the layout, treating it as equally important as the image and text modalities. Meanwhile, to alleviate the competition among modalities, we decouple the image-layout interaction into a siamese branch alongside the image-text one and fuse them in the later stage. Moreover, we contribute a large-scale layout dataset, named LayoutSAM, which includes 2.7 million image-text pairs and 10.7 million entities. Each entity is annotated with a bounding box and a detailed description. We further construct the LayoutSAM-Eval benchmark as a comprehensive tool for evaluating the L2I generation quality. Finally, we introduce the Layout Designer, which taps into the potential of large language models in layout planning, transforming them into experts in layout generation and optimization. Our code, model, and dataset will be available at <a class="link-external link-https" href="https://creatilayout.github.io" rel="external noopener nofollow">this https URL</a>.

Magic Layouts: Structural Prior for Component Detection in User Interface Designs

UI Layers Merger:Merging UI Layers Via Visual Learning and Boundary Prior

LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer

Global Beautification of 2D and 3D Layouts With Interactive Ambiguity Resolution

Layout Generation for Various Scenarios in Mobile Shopping Applications.

Creating User Interface Mock-ups from High-Level Text Descriptions with Deep-Learning Models

UI Layers Group Detector: Grouping UI Layers via Text Fusion and Box Attention

CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation

PLay: Parametrically Conditioned Layout Generation using Latent Diffusion

Gallery D.C.: Auto-created GUI Component Gallery for Design Search and Knowledge Discovery

Chat2Layout: Interactive 3D Furniture Layout with a Multimodal LLM

Automatic Layout Planning for Visually-Rich Documents with Instruction-Following Models

ULDGNN: A Fragmented UI Layer Detector Based on Graph Neural Networks

Image-aware Layout Generation with User Constraints for Poster Design

Research on Ontology-Based Usable User Interface Layout Approach

UIGuider: Detecting Implicit Design Guidelines Using a Domain Knowledge Graph Approach

UIClip: A Data-driven Model for Assessing User Interface Design

LayoutCopilot: An LLM-powered Multi-agent Collaborative Framework for Interactive Analog Layout Design

Layout-Corrector: Alleviating Layout Sticking Phenomenon in Discrete Diffusion Model

SMT-Layout: A MaxSMT-based Approach Supporting Real-time Interaction of Real-world GUI Layout

UI semantic component group detection: Grouping UI elements with similar semantics in mobile graphical user interface