Abstract:Generative adversarial networks (GANs) can be interpreted as an adversarial game between two players, a discriminator D and a generator G, in which D learns to classify real from fake data and G learns to generate realistic data by "fooling" D into thinking that fake data is actually real data. Currently, a dominating view is that G actually learns by minimizing a divergence given that the general objective function is a divergence when D is optimal. However, this view has been challenged due to inconsistencies between theory and practice. In this paper, we discuss of the properties associated with most loss functions for G (e.g., saturating/non-saturating f-GAN, LSGAN, WGAN, etc.). We show that these loss functions are not divergences and do not have the same equilibrium as expected of divergences. This suggests that G does not need to minimize the same objective function as D maximize, nor maximize the objective of D after swapping real data with fake data (non-saturating GAN) but can instead use a wide range of possible loss functions to learn to generate realistic data. We define GANs through two separate and independent D maximization and G minimization steps. We generalize the generator step to four new classes of loss functions, most of which are actual divergences (while traditional G loss functions are not). We test a wide variety of loss functions from these four classes on a synthetic dataset and on CIFAR-10. We observe that most loss functions converge well and provide comparable data generation quality to non-saturating GAN, LSGAN, and WGAN-GP generator loss functions, whether we use divergences or non-divergences. These results suggest that GANs do not conform well to the divergence minimization theory and form a much broader range of models than previously assumed.

Gradient penalty from a maximum margin perspective

Gang of GANs: Generative Adversarial Networks with Maximum Margin Ranking

Penalty Gradient Normalization for Generative Adversarial Networks

Wasserstein GANs with Gradient Penalty Compute Congested Transport

Local Stability of Wasserstein GANs With Abstract Gradient Penalty

Improved Performance of GANs via Integrating Gradient Penalty with Spectral Normalization.

GANs beyond divergence minimization

On gradient regularizers for MMD GANs

Gradient Descent Maximizes the Margin of Homogeneous Neural Networks.

The relativistic discriminator: a key element missing from standard GAN

Understanding the Effectiveness of Lipschitz Constraint in Training of GANs Via Gradient Analysis.

Local Stability and Performance of Simple Gradient Penalty mu-Wasserstein GAN

Generative Adversarial Networks with Denoising Penalty and Sample Augmentation

Towards the Gradient Vanishing, Divergence Mismatching and Mode Collapse of Generative Adversarial Nets

Improving Generalization and Stability of Generative Adversarial Networks

A framework of composite functional gradient methods for generative adversarial models

GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks

Bridging the Gap Between $f$-GANs and Wasserstein GANs

The residual generator: An improved divergence minimization framework for GAN

Discriminator contrastive divergence: Semi-amortized generative modeling by exploring energy of the discriminator