Abstract:We propose a novel ECGAN for the challenging semantic image synthesis task. Although considerable improvements have been achieved by the community in the recent period, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures; 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects); 3) Existing semantic image synthesis methods focus on modeling 'local' semantic information from a single input semantic layout. However, they ignore 'global' semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use the edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. We further propose a novel multi-scale contrastive learning method that aims to push same-class features from different scales closer together being able to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts from different scales.

Dual Attention GANs for Semantic Image Synthesis

Fine-grained Semantic Constraint in Image Synthesis

SIMGAN: Photo-Realistic Semantic Image Manipulation Using Generative Adversarial Networks.

Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis

Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis

Fine-Grained Semantic Image Synthesis with Object-Attention Generative Adversarial Network

GAN for Semantic Image Synthesis With Laplacian Pyramid and Multi-Scale Channel Attention

Dualattn-GAN: Text to Image Synthesis with Dual Attentional Generative Adversarial Network.

Remote Sensing Image Synthesis Via Semantic Embedding Generative Adversarial Networks.

DGattGAN: Cooperative Up-Sampling Based Dual Generator Attentional GAN on Text-to-Image Synthesis

Semantic Image Analogy with a Conditional Single-Image GAN

Local and Global GANs with Semantic-Aware Upsampling for Image Generation

Dual-Domain Image Synthesis using Segmentation-Guided GAN

Semantics-Enhanced Adversarial Nets for Text-to-Image Synthesis.

Multi-Channel Attention Selection GAN With Cascaded Semantic Guidance for Cross-View Image Translation

A Symmetric Dual-Attention Generative Adversarial Network with Channel and Spatial Features Fusion.

SAM-GAN: Self-Attention supporting Multi-stage Generative Adversarial Networks for text-to-image synthesis

Semantic Image Synthesis Via Diffusion Models

Asymmetric Generative Adversarial Networks with a New Attention Mechanism

Autoencoder-Based Collaborative Attention GAN for Multi-Modal Image Synthesis

Global-Affine and Local-Specific Generative Adversarial Network for Semantic-Guided Image Generation.