Abstract:We propose a novel ECGAN for the challenging semantic image synthesis task. Although considerable improvement has been achieved, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it difficult to synthesize local details and structures. 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results. 3) Existing semantic image synthesis methods focus on modeling local semantic information from a single input semantic layout. However, they ignore global semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. Doing so can capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts. Experiments on three challenging datasets show that our ECGAN achieves significantly better results than state-of-the-art methods.

SPGAN: Siamese projection Generative Adversarial Networks

SpatialGAN: Progressive Image Generation Based on Spatial Recursive Adversarial Expansion

SIMGAN: Photo-Realistic Semantic Image Manipulation Using Generative Adversarial Networks.

Creative and Diverse Artwork Generation Using Adversarial Networks

Generative Adversarial Networks with Learnable Auxiliary Module for Image Synthesis

Quality Aware Generative Adversarial Networks

Incremental Focal Loss GANs.

SP-GAN: Self-Growing and Pruning Generative Adversarial Networks

Adaptive Weighted Discriminator for Training Generative Adversarial Networks

GIU-GANs: Global Information Utilization for Generative Adversarial Networks

SGAN: An Alternative Training of Generative Adversarial Networks

Improving GAN Training via Feature Space Shrinkage

MCGAN: Enhancing GAN Training with Regression-Based Generator Loss

Towards the Gradient Vanishing, Divergence Mismatching and Mode Collapse of Generative Adversarial Nets

xAI-GAN: Enhancing Generative Adversarial Networks via Explainable AI Systems

GAN with opposition-based blocks and channel self-attention mechanism for image synthesis

On the Anomalous Generalization of GANs

SUGAN: A Stable U-Net Based Generative Adversarial Network

Various Generative Adversarial Networks Model for Synthetic Prohibitory Sign Image Generation

Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis

Smooth Deep Image Generator from Noises