Abstract:Diffusion models have been recognized for their ability to generate images that are not only visually appealing but also of high artistic quality. As a result, Layout-to-Image (L2I) generation has been proposed to leverage region-specific positions and descriptions to enable more precise and controllable generation. However, previous methods primarily focus on UNet-based models (e.g., SD1.5 and SDXL), and limited effort has explored Multimodal Diffusion Transformers (MM-DiTs), which have demonstrated powerful image generation capabilities. Enabling MM-DiT for layout-to-image generation seems straightforward but is challenging due to the complexity of how layout is introduced, integrated, and balanced among multiple modalities. To this end, we explore various network variants to efficiently incorporate layout guidance into MM-DiT, and ultimately present SiamLayout. To Inherit the advantages of MM-DiT, we use a separate set of network weights to process the layout, treating it as equally important as the image and text modalities. Meanwhile, to alleviate the competition among modalities, we decouple the image-layout interaction into a siamese branch alongside the image-text one and fuse them in the later stage. Moreover, we contribute a large-scale layout dataset, named LayoutSAM, which includes 2.7 million image-text pairs and 10.7 million entities. Each entity is annotated with a bounding box and a detailed description. We further construct the LayoutSAM-Eval benchmark as a comprehensive tool for evaluating the L2I generation quality. Finally, we introduce the Layout Designer, which taps into the potential of large language models in layout planning, transforming them into experts in layout generation and optimization. Our code, model, and dataset will be available at <a class="link-external link-https" href="https://creatilayout.github.io" rel="external noopener nofollow">this https URL</a>.

Detecting layout templates in complex multiregion files

Using Layout Information for Spreadsheet Visualization

Layout Generation for Various Scenarios in Mobile Shopping Applications.

Evaluation of a Region Proposal Architecture for Multi-task Document Layout Analysis

Image Layer Modeling for Complex Document Layout Generation.

LayoutDETR: Detection Transformer Is a Good Multimodal Layout Designer

Logical Labeling of Fixed Layout PDF Documents Using Multiple Contexts

CreatiLayout: Siamese Multimodal Diffusion Transformer for Creative Layout-to-Image Generation

A Hybrid Approach for Document Layout Analysis in Document images

WeLayout: WeChat Layout Analysis System for the ICDAR 2023 Competition on Robust Layout Segmentation in Corporate Documents

A Spreadsheet Auditing Tool Evaluated in an Industrial Context

Unified HMM-based Layout Analysis Framework and Algorithm

Semantic Table Structure Identification in Spreadsheets.

Learning to Detect Table Clones in Spreadsheets.

Synthetic document generator for annotation-free layout recognition

Visual Similarity Based Document Layout Analysis

Few-shot pixel-precise document layout segmentation via dynamic instance generation and local thresholding

M$^{6}$Doc: A Large-Scale Multi-Format, Multi-Type, Multi-Layout, Multi-Language, Multi-Annotation Category Dataset for Modern Document Layout Analysis

Tabular Structure Detection from Document Images for Resource Constrained Devices Using A Row Based Similarity Measure

Table Structure Recognition using Top-Down and Bottom-Up Cues

A Large Dataset of Historical Japanese Documents with Complex Layouts