Abstract:In this paper, we focus on the semantic image synthesis task that aims at transferring semantic label maps to photo-realistic images. Existing methods lack effective semantic constraints to preserve the semantic information and ignore the structural correlations in both spatial and channel dimensions, leading to unsatisfactory blurry and artifact-prone results. To address these limitations, we propose a novel Dual Attention GAN (DAGAN) to synthesize photo-realistic and semantically-consistent images with fine details from the input layouts without imposing extra training overhead or modifying the network architectures of existing methods. We also propose two novel modules, i.e., position-wise Spatial Attention Module (SAM) and scale-wise Channel Attention Module (CAM), to capture semantic structure attention in spatial and channel dimensions, respectively. Specifically, SAM selectively correlates the pixels at each position by a spatial attention map, leading to pixels with the same semantic label being related to each other regardless of their spatial distances. Meanwhile, CAM selectively emphasizes the scalewise features at each channel by a channel attention map, which integrates associated features among all channel maps regardless of their scales. We finally sum the outputs of SAM and CAM to further improve feature representation. Extensive experiments on four challenging datasets show that DAGAN achieves remarkably better results than state-of-the-art methods, while using fewer model parameters. The source code and trained models are available at https://github.com/Ha0Tang/DAGAN.

SCP-Diff: Photo-Realistic Semantic Image Synthesis with Spatial-Categorical Joint Prior

SCP-Diff: Spatial-Categorical Joint Prior for Diffusion Based Semantic Image Synthesis

SIMGAN: Photo-Realistic Semantic Image Manipulation Using Generative Adversarial Networks.

Fine-grained Semantic Constraint in Image Synthesis

Semantic Image Synthesis with Unconditional Generator

Semantic Probability Distribution Modeling for Diverse Semantic Image Synthesis

Diverse Semantic Image Synthesis via Probability Distribution Modeling

USIS: A unified semantic image synthesis model trained on a single or multiple samples

Dual Attention GANs for Semantic Image Synthesis

Semantic Image Synthesis via Class-Adaptive Cross-Attention

Spatial Fusion GAN for Image Synthesis

IIDM: Image-to-Image Diffusion Model for Semantic Image Synthesis

USIS: Unsupervised Semantic Image Synthesis

Controllable Face Synthesis with Semantic Latent Diffusion Models

Text-to-image Generation Based on Spatial-Channel Attention and Semantic Redescription

Stochastic Conditional Diffusion Models for Robust Semantic Image Synthesis

Learning semantic priors for texture-realistic sketch-to-image synthesis

Semantic RGB-D Image Synthesis

Semantic Image Synthesis Via Diffusion Models

Image Synthesis with Class-Aware Semantic Diffusion Models for Surgical Scene Segmentation

Semantically Multi-Modal Image Synthesis