Abstract:Generative models make huge progress to the photorealistic image synthesis in recent years. To enable humans to steer the image generation process and customize the output, many works explore the interpretable dimensions of the latent space in GANs. Existing methods edit the attributes of the output image such as orientation or color scheme by varying the latent code along certain directions. However, these methods usually require additional human annotations for each pretrained model, and they mostly focus on editing global attributes. In this work, we propose a self-supervised approach to improve the spatial steerability of GANs without searching for steerable directions in the latent space or requiring extra annotations. Specifically, we design randomly sampled Gaussian heatmaps to be encoded into the intermediate layers of generative models as spatial inductive bias. Along with training the GAN model from scratch, these heatmaps are aligned with the emerging attention of the GAN's discriminator in a self-supervised learning manner. During inference, users can interact with the spatial heatmaps in an intuitive manner, enabling them to edit the output image by adjusting the scene layout, moving, or removing objects. Moreover, we incorporate DragGAN into our framework, which facilitates fine-grained manipulation within a reasonable time and supports a coarse-to-fine editing process. Extensive experiments show that the proposed method not only enables spatial editing over human faces, animal faces, outdoor scenes, and complicated multi-object indoor scenes but also brings improvement in synthesis quality.

LatentCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions

Unsupervised Discovery of Interpretable Directions in the GAN Latent Space

NoiseCLR: A Contrastive Learning Approach for Unsupervised Discovery of Interpretable Directions in Diffusion Models

Optimal Positive Generation via Latent Transformation for Contrastive Learning

Interpreting Latent Spaces of Generative Models for Medical Images using Unsupervised Methods

Tensor Component Analysis for Interpreting the Latent Space of GANs

Unsupervised Discovery of Interpretable Directions in h-space of Pre-trained Diffusion Models

Discovering Interpretable Directions in the Semantic Latent Space of Diffusion Models

Exploring Latent Dimensions of Crowd-sourced Creativity

Learning Interpretable Representations with Informative Entanglements.

Self-Discovering Interpretable Diffusion Latent Directions for Responsible Text-to-Image Generation

GANTASTIC: GAN-based Transfer of Interpretable Directions for Disentangled Image Editing in Text-to-Image Diffusion Models

GANSpace: Discovering Interpretable GAN Controls

Unsupervised Panoptic Interpretation of Latent Spaces in GANs Using Space-Filling Vector Quantization

Spatial Steerability of GANs via Self-Supervision from Discriminator

Exploring Compositional Visual Generation with Latent Classifier Guidance

Latent Directions: A Simple Pathway to Bias Mitigation in Generative AI

Learning Disentangled Representation by Exploiting Pretrained Generative Models: A Contrastive Learning View

Interpreting the Latent Space of GANs via Correlation Analysis for Controllable Concept Manipulation

Contrastive Self-supervised Representation Learning Using Synthetic Data

Decoding Diffusion: A Scalable Framework for Unsupervised Analysis of Latent Space Biases and Representations Using Natural Language Prompts