Abstract:Recent advancements in large-scale pre-trained text-to-image models have led to remarkable progress in semantic image synthesis. Nevertheless, synthesizing high-quality images with consistent semantics and layout remains a challenge. In this paper, we propose the adaPtive LAyout-semantiC fusion modulE (PLACE) that harnesses pre-trained models to alleviate the aforementioned issues. Specifically, we first employ the layout control map to faithfully represent layouts in the feature space. Subsequently, we combine the layout and semantic features in a timestep-adaptive manner to synthesize images with realistic details. During fine-tuning, we propose the Semantic Alignment (SA) loss to further enhance layout alignment. Additionally, we introduce the Layout-Free Prior Preservation (LFP) loss, which leverages unlabeled data to maintain the priors of pre-trained models, thereby improving the visual quality and semantic consistency of synthesized images. Extensive experiments demonstrate that our approach performs favorably in terms of visual quality, semantic consistency, and layout alignment. The source code and model are available at

What problem does this paper attempt to address?

### The Problem Addressed by This Paper This paper primarily addresses several key issues in high-quality semantic image synthesis: 1. **Layout Consistency**: Maintaining semantic layout consistency during image generation is a challenge. Existing methods like ControlNet and T2I-Adapter introduce additional adapters to guide high-quality semantic image synthesis but fail to accurately integrate text semantics with the corresponding regions, resulting in layout inconsistencies in the generated results (see Figure 1). 2. **Visual Quality and Semantic Consistency**: Although some methods (such as FreestyleNet) improve semantic consistency and layout alignment through the RCA module, the reduction in feature map resolution leads to the loss of layout information. Additionally, modifying the cross-attention mechanism makes it easy to lose the prior knowledge of the pre-trained model during fine-tuning, thereby affecting visual quality and semantic consistency. To address the above issues, the authors propose a method called **PLACE** (Adaptive Layout-Semantic Fusion), which includes the following aspects: - Introducing a **Layout Control Map (LCM)** as a reliable layout representation, combined with an adaptive layout-semantic fusion module to achieve dynamic integration of layout and semantic features. - Proposing effective **Semantic Alignment Loss (SA Loss)** and **Layout-Free Prior Preservation Loss (LFP Loss)**. The former enhances the layout consistency of the generated images, while the latter utilizes existing text-image pairs to preserve the semantic prior knowledge in the pre-trained model. Through these innovations, this method not only significantly improves the visual quality and semantic consistency of the generated images but also performs well in new domains (such as synthesizing new objects, new styles, or new attributes across datasets).

PLACE: Adaptive Layout-Semantic Fusion for Semantic Image Synthesis

Fine-grained Semantic Constraint in Image Synthesis

Image Synthesis from Layout with Locality-Aware Mask Adaption

Semantic Image Synthesis with Spatially-Adaptive Normalization

Refining Synthetic Images with Semantic Layouts by Adversarial Training.

Semantic-shape Adaptive Feature Modulation for Semantic Image Synthesis

Label-free Neural Semantic Image Synthesis

Layout-Bridging Text-to-Image Synthesis

Enhancing Object Coherence in Layout-to-Image Synthesis

LayoutLLM-T2I: Eliciting Layout Guidance from LLM for Text-to-Image Generation

LoCo: Locally Constrained Training-Free Layout-to-Image Synthesis

Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis.

Interactive Image Synthesis with Panoptic Layout Generation

Training-free Composite Scene Generation for Layout-to-Image Synthesis

Freestyle Layout-to-Image Synthesis

Efficient Semantic Image Synthesis via Class-Adaptive Normalization

Layout2image: Image Generation from Layout

Semantic Palette: Guiding Scene Generation with Class Proportions

On the Diversity of Conditional Image Synthesis with Semantic Layouts.

GOAL: Grounded Text-to-image Synthesis with Joint Layout Alignment Tuning

Controllable multi-domain semantic artwork synthesis