Abstract:Despite their recent successes, generative adversarial networks (GANs) for semantic image synthesis still suffer from poor image quality when trained with only adversarial supervision. Previously, additionally employing the VGG-based perceptual loss has helped to overcome this issue, significantly improving the synthesis quality, but at the same time limited the progress of GAN models for semantic image synthesis. In this work, we propose a novel, simplified GAN model, which needs only adversarial supervision to achieve high quality results. We re-design the discriminator as a semantic segmentation network, directly using the given semantic label maps as the ground truth for training. By providing stronger supervision to the discriminator as well as to the generator through spatially- and semantically-aware discriminator feedback, we are able to synthesize images of higher fidelity and with a better alignment to their input label maps, making the use of the perceptual loss superfluous. Furthermore, we enable high-quality multi-modal image synthesis through global and local sampling of a 3D noise tensor injected into the generator, which allows complete or partial image editing. We show that images synthesized by our model are more diverse and follow the color and texture distributions of real images more closely. We achieve a strong improvement in image synthesis quality over prior state-of-the-art models across the commonly used ADE20K, Cityscapes, and COCO-Stuff datasets using only adversarial supervision. In addition, we investigate semantic image synthesis under severe class imbalance and sparse annotations, which are common aspects in practical applications but were overlooked in prior works. To this end, we evaluate our model on LVIS, a dataset originally introduced for long-tailed object recognition. We thereby demonstrate high performance of our model in the sparse and unbalanced data regimes, achieved by means of the proposed 3D noise and the ability of our discriminator to balance class contributions directly in the loss function. Our code and pretrained models are available at https://github.com/boschresearch/OASIS.

USIS: Unsupervised Semantic Image Synthesis

USIS: A unified semantic image synthesis model trained on a single or multiple samples

Wavelet-based Unsupervised Label-to-Image Translation

Semantic Image Synthesis with Unconditional Generator

Towards Pragmatic Semantic Image Synthesis for Urban Scenes

Semantic Image Synthesis via Class-Adaptive Cross-Attention

Diverse Semantic Image Synthesis with Various Conditioning Modalities

SCP-Diff: Photo-Realistic Semantic Image Synthesis with Spatial-Categorical Joint Prior

Semantically Multi-Modal Image Synthesis

Unpaired semantic neural person image synthesis

Synthetic Instance Segmentation from Semantic Image Segmentation Masks

Image Synthesis with Class-Aware Semantic Diffusion Models for Surgical Scene Segmentation

OASIS: Only Adversarial Supervision for Semantic Image Synthesis

SIEDOB: Semantic Image Editing by Disentangling Object and Background

SVS-GAN: Leveraging GANs for Semantic Video Synthesis

Controllable Face Synthesis with Semantic Latent Diffusion Models

Semantic RGB-D Image Synthesis

Artistic Image Synthesis from Unsupervised Segmentation Maps

Unlocking Pre-trained Image Backbones for Semantic Image Synthesis

Remote Sensing Image Synthesis Via Semantic Embedding Generative Adversarial Networks.

Weakly Supervised Semantic Segmentation in Aerial Imagery via Cross-Image Semantic Mining