Abstract:Existing generative adversarial network (GAN) based conditional imagegenerative models typically produce fixed output for the same conditionalinput, which is unreasonable for highly subjective tasks, such as large-maskimage inpainting or style transfer. On the other hand, GAN-based diverse imagegenerative methods require retraining/fine-tuning the network or designingcomplex noise injection functions, which is computationally expensive,task-specific, or struggle to generate high-quality results. Given that manydeterministic conditional image generative models have been able to producehigh-quality yet fixed results, we raise an intriguing question: is it possiblefor pre-trained deterministic conditional image generative models to generatediverse results without changing network structures or parameters? To answerthis question, we re-examine the conditional image generation tasks from theperspective of adversarial attack and propose a simple and efficient plug-inprojected gradient descent (PGD) like method for diverse and controllable imagegeneration. The key idea is attacking the pre-trained deterministic generativemodels by adding a micro perturbation to the input condition. In this way,diverse results can be generated without any adjustment of network structuresor fine-tuning of the pre-trained models. In addition, we can also control thediverse results to be generated by specifying the attack direction according toa reference text or image. Our work opens the door to applying adversarialattack to low-level vision tasks, and experiments on various conditional imagegeneration tasks demonstrate the effectiveness and superiority of the proposedmethod.

Conditional Generative Adversarial Nets

Attack Deterministic Conditional Image Generative Models for Diverse and Controllable Generation

Ways of Conditioning Generative Adversarial Networks

Statistics Enhancement Generative Adversarial Networks for Diverse Conditional Image Synthesis

Conditional Idempotent Generative Networks

Generative Adversarial Nets

Sequence Generative Adversarial Nets with a Conditional Discriminator

Guiding GANs: How to control non-conditional pre-trained GANs for conditional image generation

Virtual Conditional Generative Adversarial Networks

A one-to-many conditional generative adversarial network framework for multiple image-to-image translations

Structured Generative Adversarial Networks.

Conditional Generation from Unconditional Diffusion Models using Denoiser Representations

Multi-Label Conditional Generation From Pre-Trained Models

Conditional Image Synthesis With Auxiliary Classifier GANs

Conditional Image Generation with PixelCNN Decoders

Text to image synthesis using multi-generator text conditioned generative adversarial networks.

Generative adversarial networks

Conditional Information Gain Networks

Triple Generative Adversarial Networks.

Multi-class Generative Adversarial Nets for Semi-supervised Image Classification

Antimode collapse generative adversarial networks