Abstract:We propose a novel ECGAN for the challenging semantic image synthesis task. Although considerable improvement has been achieved, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it difficult to synthesize local details and structures. 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results. 3) Existing semantic image synthesis methods focus on modeling local semantic information from a single input semantic layout. However, they ignore global semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. Edge information is produced by a convolutional generator and introduces detailed structure information. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. Doing so can capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts. Experiments on three challenging datasets show that our ECGAN achieves significantly better results than state-of-the-art methods.

Enhancing GANs with Contrastive Learning-Based Multistage Progressive Finetuning SNN and RL-Based External Optimization

OA-GAN: Organ-Aware Generative Adversarial Network for Synthesizing Contrast-Enhanced Medical Images

RIRGAN: an End-to-end Lightweight Multi-Task Learning Method for Brain MRI Super-Resolution and Denoising

A Mixed-Supervision Multilevel GAN Framework for Image Quality Enhancement

GL-GAN: Adaptive Global and Local Bilevel Optimization model of Image Generation

FS-GAN: Fuzzy Self-guided Structure Retention Generative Adversarial Network for Medical Image Enhancement

GANetic Loss for Generative Adversarial Networks with a Focus on Medical Applications

Improving GAN with Neighbors Embedding and Gradient Matching

MCGAN: Enhancing GAN Training with Regression-Based Generator Loss

High-resolution medical image synthesis using progressively grown generative adversarial networks

Enhanced Magnetic Resonance Image Synthesis with Contrast-Aware Generative Adversarial Networks

InfoMax-GAN: Improved Adversarial Image Generation via Information Maximization and Contrastive Learning

MedGAN: An adaptive GAN approach for medical image generation

Progressive Growing of GANs for Improved Quality, Stability, and Variation

Remote Sensing Image Super-Resolution via Saliency-Guided Feedback GANs

GACnet-Text-to-Image Synthesis With Generative Models Using Attention Mechanisms With Contrastive Learning

Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis

Fine-tuned Generative Adversarial Network-based Model for Medical Image Super-Resolution

Progressive Generative Adversarial Networks for Medical Image Super resolution

GANs for Medical Image Synthesis: An Empirical Study

A Self-attention Guided Multi-scale Gradient GAN for Diversified X-ray Image Synthesis