Abstract:We propose a novel ECGAN for the challenging semantic image synthesis task. Although considerable improvements have been achieved by the community in the recent period, the quality of synthesized images is far from satisfactory due to three largely unresolved challenges. 1) The semantic labels do not provide detailed structural information, making it challenging to synthesize local details and structures; 2) The widely adopted CNN operations such as convolution, down-sampling, and normalization usually cause spatial resolution loss and thus cannot fully preserve the original semantic information, leading to semantically inconsistent results (e.g., missing small objects); 3) Existing semantic image synthesis methods focus on modeling 'local' semantic information from a single input semantic layout. However, they ignore 'global' semantic information of multiple input semantic layouts, i.e., semantic cross-relations between pixels across different input layouts. To tackle 1), we propose to use the edge as an intermediate representation which is further adopted to guide image generation via a proposed attention guided edge transfer module. To tackle 2), we design an effective module to selectively highlight class-dependent feature maps according to the original semantic layout to preserve the semantic information. To tackle 3), inspired by current methods in contrastive learning, we propose a novel contrastive learning method, which aims to enforce pixel embeddings belonging to the same semantic class to generate more similar image content than those from different classes. We further propose a novel multi-scale contrastive learning method that aims to push same-class features from different scales closer together being able to capture more semantic relations by explicitly exploring the structures of labeled pixels from multiple input semantic layouts from different scales.

Synthetic Convolutional Features for Improved Semantic Segmentation

Fine-grained Semantic Constraint in Image Synthesis

FreeMask: Synthetic Images with Dense Annotations Make Stronger Segmentation Models

Research of improving semantic image segmentation based on a feature fusion model

Learning Semantic Segmentation from Synthetic Data: A Geometrically Guided Input-Output Adaptation Approach

Semantic Image Synthesis with Unconditional Generator

Exploring Limits of Diffusion-Synthetic Training with Weakly Supervised Semantic Segmentation

Improving Semantic Segmentation of Urban Scenes for Self-Driving Cars with Synthetic Images

Few-shot Semantic Image Synthesis with Class Affinity Transfer

Exploiting Semantics in Adversarial Training for Image-Level Domain Adaptation

Learning to Predict Layout-to-image Conditional Convolutions for Semantic Image Synthesis.

Fully Convolutional Adaptation Networks for Semantic Segmentation

Unlocking Pre-trained Image Backbones for Semantic Image Synthesis

Edge Guided GANs with Contrastive Learning for Semantic Image Synthesis

ScribbleGen: Generative Data Augmentation Improves Scribble-supervised Semantic Segmentation

Semantic Segmentation by Improved Generative Adversarial Networks

Learning semantic priors for texture-realistic sketch-to-image synthesis

Image Synthesis with Class-Aware Semantic Diffusion Models for Surgical Scene Segmentation

Improving Synthetic to Realistic Semantic Segmentation with Parallel Generative Ensembles for Autonomous Urban Driving

Edge Guided GANs with Multi-Scale Contrastive Learning for Semantic Image Synthesis

Synthetic dual image generation for reduction of labeling efforts in semantic segmentation of micrographs with a customized metric function