Abstract:Generative adversarial networks (GANs) can be interpreted as an adversarial game between two players, a discriminator D and a generator G, in which D learns to classify real from fake data and G learns to generate realistic data by "fooling" D into thinking that fake data is actually real data. Currently, a dominating view is that G actually learns by minimizing a divergence given that the general objective function is a divergence when D is optimal. However, this view has been challenged due to inconsistencies between theory and practice. In this paper, we discuss of the properties associated with most loss functions for G (e.g., saturating/non-saturating f-GAN, LSGAN, WGAN, etc.). We show that these loss functions are not divergences and do not have the same equilibrium as expected of divergences. This suggests that G does not need to minimize the same objective function as D maximize, nor maximize the objective of D after swapping real data with fake data (non-saturating GAN) but can instead use a wide range of possible loss functions to learn to generate realistic data. We define GANs through two separate and independent D maximization and G minimization steps. We generalize the generator step to four new classes of loss functions, most of which are actual divergences (while traditional G loss functions are not). We test a wide variety of loss functions from these four classes on a synthetic dataset and on CIFAR-10. We observe that most loss functions converge well and provide comparable data generation quality to non-saturating GAN, LSGAN, and WGAN-GP generator loss functions, whether we use divergences or non-divergences. These results suggest that GANs do not conform well to the divergence minimization theory and form a much broader range of models than previously assumed.

A framework of composite functional gradient methods for generative adversarial models

A Framework of Composite Functional Gradient Methods for Generative Adversarial Models

Training Generative Adversarial Networks with Adaptive Composite Gradient

A New Distributed Method for Training Generative Adversarial Networks

A Convex Duality Framework for GANs

Many paths to equilibrium: GANs do not need to decrease a divergence at every step

A Novel Framework for Selection of GANs for an Application

Towards the Gradient Vanishing, Divergence Mismatching and Mode Collapse of Generative Adversarial Nets

GANs beyond divergence minimization

The residual generator: An improved divergence minimization framework for GAN

Local Convergence of Gradient Descent-Ascent for Training Generative Adversarial Networks

Generative adversarial networks

GEGAN: Generative Adversarial Networks Training Based on Gradient Directed Evolutionary Strategy

Online Kernel based Generative Adversarial Networks

An Online Learning Approach to Generative Adversarial Networks

Generative Adversarial Networks: The State of the Art and Beyond

A New Paradigm for Generative Adversarial Networks based on Randomized Decision Rules

Evaluation of GAN-Based Model for Adversarial Training

The theoretical research of generative adversarial networks: an overview

Effective Dynamics of Generative Adversarial Networks

Generative Adversarial Nets