SSMG: Spatial-Semantic Map Guided Diffusion Model for Free-form Layout-to-Image Generation

Chengyou Jia,Minnan Luo,Zhuohang Dang,Guang Dai,Xiaojun Chang,Mengmeng Wang,Jingdong Wang

2024-03-13

Abstract:Despite significant progress in Text-to-Image (T2I) generative models, even lengthy and complex text descriptions still struggle to convey detailed controls. In contrast, Layout-to-Image (L2I) generation, aiming to generate realistic and complex scene images from user-specified layouts, has risen to prominence. However, existing methods transform layout information into tokens or RGB images for conditional control in the generative process, leading to insufficient spatial and semantic controllability of individual instances. To address these limitations, we propose a novel Spatial-Semantic Map Guided (SSMG) diffusion model that adopts the feature map, derived from the layout, as guidance. Owing to rich spatial and semantic information encapsulated in well-designed feature maps, SSMG achieves superior generation quality with sufficient spatial and semantic controllability compared to previous works. Additionally, we propose the Relation-Sensitive Attention (RSA) and Location-Sensitive Attention (LSA) mechanisms. The former aims to model the relationships among multiple objects within scenes while the latter is designed to heighten the model's sensitivity to the spatial information embedded in the guidance. Extensive experiments demonstrate that SSMG achieves highly promising results, setting a new state-of-the-art across a range of metrics encompassing fidelity, diversity, and controllability.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the shortcomings of Text-to-Image (T2I) generation models in detailed control and proposes a new Layout-to-Image (L2I) generation method. Although T2I models have made significant progress, even long and complex text descriptions find it difficult to precisely control the details in the generated images. In contrast, the L2I generation method creates realistic and complex scene images through user-specified layouts, offering higher fine-grained control. However, existing L2I methods typically convert layout information into labels or RGB images to achieve conditional control during the generation process, leading to insufficient spatial and semantic control over individual instances.

To address the above limitations, the authors propose a novel Spatial-Semantic Map Guided diffusion model (SSMG), which uses feature maps derived from the layout as guiding signals. Since well-designed feature maps contain rich spatial and semantic information, SSMG outperforms previous works in generation quality, offering better spatial and semantic controllability. Additionally, the authors introduce Relation-Sensitive Attention (RSA) and Location-Sensitive Attention (LSA). The former is used to model the relationships between multiple objects within a scene, while the latter enhances the model's sensitivity to embedded spatial information. Experimental results show that SSMG performs excellently across a range of evaluation metrics, including fidelity, diversity, and controllability, with a significant improvement in YOLO scores, thereby establishing a new technological frontier.