From GAN to WGAN

Lilian Weng

DOI: https://doi.org/10.48550/arXiv.1904.08994

IF: 5.414

2019-04-18

Machine Learning

Abstract:This paper explains the math behind a generative adversarial network (GAN) model and why it is hard to be trained. Wasserstein GAN is intended to improve GANs' training by adopting a smooth metric for measuring the distance between two probability distributions.

What problem does this paper attempt to address?

Some Theoretical Insights into Wasserstein GANs

Gérard Biau,Maxime Sangnier,Ugo Tanielian

DOI: https://doi.org/10.48550/arXiv.2006.02682

2021-06-18

Abstract:Generative Adversarial Networks (GANs) have been successful in producing outstanding results in areas as diverse as image, video, and text generation. Building on these successes, a large number of empirical studies have validated the benefits of the cousin approach called Wasserstein GANs (WGANs), which brings stabilization in the training process. In the present paper, we add a new stone to the edifice by proposing some theoretical advances in the properties of WGANs. First, we properly define the architecture of WGANs in the context of integral probability metrics parameterized by neural networks and highlight some of their basic mathematical features. We stress in particular interesting optimization properties arising from the use of a parametric 1-Lipschitz discriminator. Then, in a statistically-driven approach, we study the convergence of empirical WGANs as the sample size tends to infinity, and clarify the adversarial effects of the generator and the discriminator by underlining some trade-off properties. These features are finally illustrated with experiments using both synthetic and real-world datasets.

Machine Learning
Wasserstein Generative Adversarial Networks

Soumith Chintala,Martín Arjovsky,L. Bottou

2017-07-17

Abstract:We introduce a new algorithm named WGAN, an alternative to traditional GAN training. In this new model, we show that we can improve the stability of learning, get rid of problems like mode collapse, and provide meaningful learning curves useful for debugging and hyperparameter searches. Furthermore, we show that the corresponding optimization problem is sound, and provide extensive theoretical work highlighting the deep connections to different distances between distributions.

Computer Science,Mathematics
A Wasserstein perspective of Vanilla GANs

Lea Kunkel,Mathias Trabs

2024-07-29

Abstract:The empirical success of Generative Adversarial Networks (GANs) caused an increasing interest in theoretical research. The statistical literature is mainly focused on Wasserstein GANs and generalizations thereof, which especially allow for good dimension reduction properties. Statistical results for Vanilla GANs, the original optimization problem, are still rather limited and require assumptions such as smooth activation functions and equal dimensions of the latent space and the ambient space. To bridge this gap, we draw a connection from Vanilla GANs to the Wasserstein distance. By doing so, existing results for Wasserstein GANs can be extended to Vanilla GANs. In particular, we obtain an oracle inequality for Vanilla GANs in Wasserstein distance. The assumptions of this oracle inequality are designed to be satisfied by network architectures commonly used in practice, such as feedforward ReLU networks. By providing a quantitative result for the approximation of a Lipschitz function by a feedforward ReLU network with bounded Hölder norm, we conclude a rate of convergence for Vanilla GANs as well as Wasserstein GANs as estimators of the unknown probability distribution.

Statistics Theory,Machine Learning
Optimal 1-Wasserstein Distance for WGANs

Arthur Stéphanovitch,Ugo Tanielian,Benoît Cadre,Nicolas Klutchnikoff,Gérard Biau

2023-10-05

Abstract:The mathematical forces at work behind Generative Adversarial Networks raise challenging theoretical issues. Motivated by the important question of characterizing the geometrical properties of the generated distributions, we provide a thorough analysis of Wasserstein GANs (WGANs) in both the finite sample and asymptotic regimes. We study the specific case where the latent space is univariate and derive results valid regardless of the dimension of the output space. We show in particular that for a fixed sample size, the optimal WGANs are closely linked with connected paths minimizing the sum of the squared Euclidean distances between the sample points. We also highlight the fact that WGANs are able to approach (for the 1-Wasserstein distance) the target distribution as the sample size tends to infinity, at a given convergence rate and provided the family of generative Lipschitz functions grows appropriately. We derive in passing new results on optimal transport theory in the semi-discrete setting.

Machine Learning,Statistics Theory
Relaxed Wasserstein with Applications to GANs

Xin Guo,Johnny Hong,Tianyi Lin,Nan Yang

DOI: https://doi.org/10.48550/arXiv.1705.07164

2021-07-17

Abstract:Wasserstein Generative Adversarial Networks (WGANs) provide a versatile class of models, which have attracted great attention in various applications. However, this framework has two main drawbacks: (i) Wasserstein-1 (or Earth-Mover) distance is restrictive such that WGANs cannot always fit data geometry well; (ii) It is difficult to achieve fast training of WGANs. In this paper, we propose a new class of \textit{Relaxed Wasserstein} (RW) distances by generalizing Wasserstein-1 distance with Bregman cost functions. We show that RW distances achieve nice statistical properties while not sacrificing the computational tractability. Combined with the GANs framework, we develop Relaxed WGANs (RWGANs) which are not only statistically flexible but can be approximated efficiently using heuristic approaches. Experiments on real images demonstrate that the RWGAN with Kullback-Leibler (KL) cost function outperforms other competing approaches, e.g., WGANs, even with gradient penalty.

Machine Learning
Wasserstein GANs with Gradient Penalty Compute Congested Transport

Tristan Milne,Adrian Nachman

DOI: https://doi.org/10.48550/arXiv.2109.00528

2022-07-01

Abstract:Wasserstein GANs with Gradient Penalty (WGAN-GP) are a very popular method for training generative models to produce high quality synthetic data. While WGAN-GP were initially developed to calculate the Wasserstein 1 distance between generated and real data, recent works (e.g. [23]) have provided empirical evidence that this does not occur, and have argued that WGAN-GP perform well not in spite of this issue, but because of it. In this paper we show for the first time that WGAN-GP compute the minimum of a different optimal transport problem, the so-called congested transport [7]. Congested transport determines the cost of moving one distribution to another under a transport model that penalizes congestion. For WGAN-GP, we find that the congestion penalty has a spatially varying component determined by the sampling strategy used in [12] which acts like a local speed limit, making congestion cost less in some regions than others. This aspect of the congested transport problem is new, in that the congestion penalty turns out to be unbounded and depends on the distributions to be transported, and so we provide the necessary mathematical proofs for this setting. One facet of our discovery is a formula connecting the gradient of solutions to the optimization problem in WGAN-GP to the time averaged momentum of the optimal mass flow. This is in contrast to the gradient of Kantorovich potentials for the Wasserstein 1 distance, which is just the normalized direction of flow. Based on this and other considerations, we speculate on how our results explain the observed performance of WGAN-GP. Beyond applications to GANs, our theorems also point to the possibility of approximately solving large scale congested transport problems using neural network techniques.

Machine Learning,Optimization and Control
Understanding the Effectiveness of Lipschitz Constraint in Training of GANs Via Gradient Analysis.

Zhiming Zhou,Yuxuan Song,Lantao Yu,Yong Yu

2018-01-01

Abstract:In this paper, we investigate the underlying factor that leads to failure and success in the training of GANs. We study the property of the optimal discriminative function and show that in many GANs, the gradient from the optimal discriminative function is not reliable, which turns out to be the fundamental cause of failure in training of GANs. We further demonstrate that a well-defined distance metric does not necessarily guarantee the convergence of GANs. Finally, we prove in this paper that Lipschitz-continuity condition is a general solution to make the gradient of the optimal discriminative function reliable, and characterized the necessary condition where Lipschitz-continuity ensures the convergence, which leads to a broad family of valid GAN objectives under Lipschitz-continuity condition, where Wasserstein distance is one special case. We experiment with several new objectives, which are sound according to our theorems, and we found that, compared with Wasserstein distance, the outputs of the discriminator with new objectives are more stable and the final qualities of generated samples are also consistently higher than those produced by Wasserstein distance.
Inferential Wasserstein generative adversarial networks

Yao Chen,Qingyi Gao,Xiao Wang

DOI: https://doi.org/10.1111/rssb.12476

2021-11-15

Abstract:Abstract Generative adversarial networks (GANs) have been impactful on many problems and applications but suffer from unstable training. The Wasserstein GAN (WGAN) leverages the Wasserstein distance to avoid the caveats in the minmax two-player training of GANs but has other defects such as mode collapse and lack of metric to detect the convergence. We introduce a novel inferential Wasserstein GAN (iWGAN) model, which is a principled framework to fuse autoencoders and WGANs. The iWGAN model jointly learns an encoder network and a generator network motivated by the iterative primal-dual optimization process. The encoder network maps the observed samples to the latent space and the generator network maps the samples from the latent space to the data space. We establish the generalization error bound of the iWGAN to theoretically justify its performance. We further provide a rigorous probabilistic interpretation of our model under the framework of maximum likelihood estimation. The iWGAN, with a clear stopping criteria, has many advantages over other autoencoder GANs. The empirical experiments show that the iWGAN greatly mitigates the symptom of mode collapse, speeds up the convergence, and is able to provide a measurement of quality check for each individual sample. We illustrate the ability of the iWGAN by obtaining competitive and stable performances for benchmark datasets.

statistics & probability
Local Stability of Wasserstein GANs With Abstract Gradient Penalty

Cheolhyeong Kim,Seungtae Park,Hyung Ju Hwang

DOI: https://doi.org/10.1109/tnnls.2021.3057885

IF: 14.255

2021-01-01

IEEE Transactions on Neural Networks and Learning Systems

Abstract:The convergence of generative adversarial networks (GANs) has been studied substantially in various aspects to achieve successful generative tasks. Ever since it is first proposed, the idea has achieved many theoretical improvements by injecting an instance noise, choosing different divergences, penalizing the discriminator, and so on. In essence, these efforts are to approximate a real-world measure with an idle measure through a learning procedure. In this article, we provide an analysis of GANs in the most general setting to reveal what, in essence, should be satisfied to achieve successful convergence. This work is not trivial since handling a converging sequence of an abstract measure requires a lot more sophisticated concepts. In doing so, we find an interesting fact that the discriminator can be penalized in a more general setting than what has been implemented. Furthermore, our experiment results substantiate our theoretical argument on various generative tasks.

computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture
Language Modeling with Generative Adversarial Networks

Mehrad Moradshahi,Utkarsh Contractor

DOI: https://doi.org/10.48550/arXiv.1804.02617

IF: 5.414

2018-04-08

Machine Learning

Abstract:Generative Adversarial Networks (GANs) have been promising in the field of image generation, however, they have been hard to train for language generation. GANs were originally designed to output differentiable values, so discrete language generation is challenging for them which causes high levels of instability in training GANs. Consequently, past work has resorted to pre-training with maximum-likelihood or training GANs without pre-training with a WGAN objective with a gradient penalty. In this study, we present a comparison of those approaches. Furthermore, we present the results of some experiments that indicate better training and convergence of Wasserstein GANs (WGANs) when a weaker regularization term is enforcing the Lipschitz constraint.
On the Convergence and Robustness of Training GANs with Regularized Optimal Transport

Maziar Sanjabi,Jimmy Ba,Meisam Razaviyayn,Jason D. Lee

DOI: https://doi.org/10.48550/arXiv.1802.08249

IF: 5.414

2018-02-22

Machine Learning

Abstract:Generative Adversarial Networks (GANs) are one of the most practical methods for learning data distributions. A popular GAN formulation is based on the use of Wasserstein distance as a metric between probability distributions. Unfortunately, minimizing the Wasserstein distance between the data distribution and the generative model distribution is a computationally challenging problem as its objective is non-convex, non-smooth, and even hard to compute. In this work, we show that obtaining gradient information of the smoothed Wasserstein GAN formulation, which is based on regularized Optimal Transport (OT), is computationally effortless and hence one can apply first order optimization methods to minimize this objective. Consequently, we establish theoretical convergence guarantee to stationarity for a proposed class of GAN optimization algorithms. Unlike the original non-smooth formulation, our algorithm only requires solving the discriminator to approximate optimality. We apply our method to learning MNIST digits as well as CIFAR-10images. Our experiments show that our method is computationally efficient and generates images comparable to the state of the art algorithms given the same architecture and computational power.
A Wasserstein GAN model with the total variational regularization

Lijun Zhang,Yujin Zhang,Yongbin Gao

DOI: https://doi.org/10.48550/arXiv.1812.00810

2018-12-03

Computer Vision and Pattern Recognition

Abstract:It is well known that the generative adversarial nets (GANs) are remarkably difficult to train. The recently proposed Wasserstein GAN (WGAN) creates principled research directions towards addressing these issues. But we found in practice that gradient penalty WGANs (GP-WGANs) still suffer from training instability. In this paper, we combine a Total Variational (TV) regularizing term into the WGAN formulation instead of weight clipping or gradient penalty, which implies that the Lipschitz constraint is enforced on the critic network. Our proposed method is more stable at training than GP-WGANs and works well across varied GAN architectures. We also present a method to control the trade-off between image diversity and visual quality. It does not bring any computation burden.
(q,p)-Wasserstein GANs: Comparing Ground Metrics for Wasserstein GANs

Anton Mallasto,Jes Frellsen,Wouter Boomsma,Aasa Feragen

DOI: https://doi.org/10.48550/arXiv.1902.03642

2019-02-11

Abstract:Generative Adversial Networks (GANs) have made a major impact in computer vision and machine learning as generative models. Wasserstein GANs (WGANs) brought Optimal Transport (OT) theory into GANs, by minimizing the $1$-Wasserstein distance between model and data distributions as their objective function. Since then, WGANs have gained considerable interest due to their stability and theoretical framework. We contribute to the WGAN literature by introducing the family of $(q,p)$-Wasserstein GANs, which allow the use of more general $p$-Wasserstein metrics for $p\geq 1$ in the GAN learning procedure. While the method is able to incorporate any cost function as the ground metric, we focus on studying the $l^q$ metrics for $q\geq 1$. This is a notable generalization as in the WGAN literature the OT distances are commonly based on the $l^2$ ground metric. We demonstrate the effect of different $p$-Wasserstein distances in two toy examples. Furthermore, we show that the ground metric does make a difference, by comparing different $(q,p)$ pairs on the MNIST and CIFAR-10 datasets. Our experiments demonstrate that changing the ground metric and $p$ can notably improve on the common $(q,p) = (2,1)$ case.

Machine Learning
Optimizing Latent Distributions for Non-Adversarial Generative Networks

Tianyu Guo,Chang Xu,Boxin Shi,Chao Xu,Dacheng Tao

DOI: https://doi.org/10.1109/tpami.2020.3043745

IF: 23.6

2020-01-01

IEEE Transactions on Pattern Analysis and Machine Intelligence

Abstract:The generator in generative adversarial networks (GANs) is driven by a discriminator to produce high-quality images through an adversarial game. At the same time, the difficulty of reaching a stable generator has been increased. This paper focuses on non-adversarial generative networks that are trained in a plain manner without adversarial loss. The given limited number of real images could be insufficient to fully represent the real data distribution. We therefore investigate a set of distributions in a Wasserstein ball centred on the distribution induced by the training data and propose to optimize the generator over this Wasserstein ball. We theoretically discuss the solvability of the newly defined objective function and develop a tractable reformulation to learn the generator. The connections and differences between the proposed non-adversarial generative networks and GANs are analyzed. Experimental results on real-world datasets demonstrate that the proposed algorithm can effectively learn image generators in a non-adversarial approach, and the generated images are of comparable quality with those from GANs.
Adversarial network training using higher-order moments in a modified Wasserstein distance

Oliver Serang

DOI: https://doi.org/10.48550/arXiv.2210.03354

2022-10-07

Abstract:Generative-adversarial networks (GANs) have been used to produce data closely resembling example data in a compressed, latent space that is close to sufficient for reconstruction in the original vector space. The Wasserstein metric has been used as an alternative to binary cross-entropy, producing more numerically stable GANs with greater mode covering behavior. Here, a generalization of the Wasserstein distance, using higher-order moments than the mean, is derived. Training a GAN with this higher-order Wasserstein metric is demonstrated to exhibit superior performance, even when adjusted for slightly higher computational cost. This is illustrated generating synthetic antibody sequences.

Machine Learning
Soft Generative Adversarial Network: Combating Mode Collapse in Generative Adversarial Network Training Via Dynamic Borderline Softening Mechanism

Wei Li,Yongchuan Tang

DOI: https://doi.org/10.3390/app14020579

2024-01-01

Abstract:In this paper, we propose the Soft Generative Adversarial Network (SoftGAN), a strategy that utilizes a dynamic borderline softening mechanism to train Generative Adversarial Networks. This mechanism aims to solve the mode collapse problem and enhance the training stability of the generated outputs. Within the SoftGAN, the objective of the discriminator is to learn a fuzzy concept of real data with a soft borderline between real and generated data. This objective is achieved by balancing the principles of maximum concept coverage and maximum expected entropy of fuzzy concepts. During the early training stage of the SoftGAN, the principle of maximum expected entropy of fuzzy concepts guides the learning process due to the significant divergence between the generated and real data. However, in the final stage of training, the principle of maximum concept coverage dominates as the divergence between the two distributions decreases. The dynamic borderline softening mechanism of the SoftGAN can be likened to a student (the generator) striving to create realistic images, with the tutor (the discriminator) dynamically guiding the student towards the right direction and motivating effective learning. The tutor gives appropriate encouragement or requirements according to abilities of the student at different stages, so as to promote the student to improve themselves better. Our approach offers both theoretical and practical benefits for improving GAN training. We empirically demonstrate the superiority of our SoftGAN approach in addressing mode collapse issues and generating high-quality outputs compared to existing approaches.
Generative Adversarial Network: Some Analytical Perspectives

Haoyang Cao,Xin Guo

DOI: https://doi.org/10.48550/arXiv.2104.12210

2021-04-25

Mathematical Finance

Abstract:Ever since its debut, generative adversarial networks (GANs) have attracted tremendous amount of attention. Over the past years, different variations of GANs models have been developed and tailored to different applications in practice. Meanwhile, some issues regarding the performance and training of GANs have been noticed and investigated from various theoretical perspectives. This subchapter will start from an introduction of GANs from an analytical perspective, then move on to the training of GANs via SDE approximations and finally discuss some applications of GANs in computing high dimensional MFGs as well as tackling mathematical finance problems.
A Mathematical Introduction to Generative Adversarial Nets (GAN)

Yang Wang

DOI: https://doi.org/10.48550/arXiv.2009.00169

IF: 5.414

2020-09-01

Machine Learning

Abstract:Generative Adversarial Nets (GAN) have received considerable attention since the 2014 groundbreaking work by Goodfellow et al. Such attention has led to an explosion in new ideas, techniques and applications of GANs. To better understand GANs we need to understand the mathematical foundation behind them. This paper attempts to provide an overview of GANs from a mathematical point of view. Many students in mathematics may find the papers on GANs more difficulty to fully understand because most of them are written from computer science and engineer point of view. The aim of this paper is to give more mathematically oriented students an introduction to GANs in a language that is more familiar to them.

From GAN to WGAN

Some Theoretical Insights into Wasserstein GANs

Wasserstein Generative Adversarial Networks

A Wasserstein perspective of Vanilla GANs

Optimal 1-Wasserstein Distance for WGANs

Relaxed Wasserstein with Applications to GANs

Wasserstein GANs with Gradient Penalty Compute Congested Transport

Understanding the Effectiveness of Lipschitz Constraint in Training of GANs Via Gradient Analysis.

Inferential Wasserstein generative adversarial networks

Local Stability of Wasserstein GANs With Abstract Gradient Penalty

Language Modeling with Generative Adversarial Networks

On the Convergence and Robustness of Training GANs with Regularized Optimal Transport

A Wasserstein GAN model with the total variational regularization

(q,p)-Wasserstein GANs: Comparing Ground Metrics for Wasserstein GANs

Optimizing Latent Distributions for Non-Adversarial Generative Networks

Adversarial network training using higher-order moments in a modified Wasserstein distance

Soft Generative Adversarial Network: Combating Mode Collapse in Generative Adversarial Network Training Via Dynamic Borderline Softening Mechanism

Generative Adversarial Network: Some Analytical Perspectives

A Mathematical Introduction to Generative Adversarial Nets (GAN)