Abstract:Despite the success of Lipschitz regularization in stabilizing GAN training, the exact reason of its effectiveness remains poorly understood. The direct effect of K-Lipschitz regularization is to restrict the L2-norm of the neural network gradient to be smaller than a threshold K (e.g., K=1\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$K=1$$\end{document}) such that ‖∇f‖≤K\documentclass[12pt]{minimal}\usepackage{amsmath}\usepackage{wasysym}\usepackage{amsfonts}\usepackage{amssymb}\usepackage{amsbsy}\usepackage{mathrsfs}\usepackage{upgreek}\setlength{\oddsidemargin}{-69pt}\begin{document}$$\Vert \nabla f\Vert \le K$$\end{document}. In this work, we uncover an even more important effect of Lipschitz regularization by examining its impact on the loss function: It degenerates GAN loss functions to almost linear ones by restricting their domain and interval of attainable gradient values. Our analysis shows that loss functions are only successful if they are degenerated to almost linear ones. We also show that loss functions perform poorly if they are not degenerated and that a wide range of functions can be used as loss function as long as they are sufficiently degenerated by regularization. Basically, Lipschitz regularization ensures that all loss functions effectively work in the same way. Empirically, we verify our proposition on the MNIST, CIFAR10 and CelebA datasets.

Lipschitz Constrained GANs via Boundedness and Continuity

Understanding the Effectiveness of Lipschitz Constraint in Training of GANs Via Gradient Analysis.

Towards Efficient and Unbiased Implementation of Lipschitz Continuity in GANs

Understanding the Effectiveness of Lipschitz-Continuity in Generative Adversarial Nets

Lipschitz Generative Adversarial Nets

Effective Lipschitz constraint enforcement for Wasserstein GAN training

Spectral bounding: Strictly satisfying the 1-Lipschitz property for generative adversarial networks

CHAIN: Enhancing Generalization in Data-Efficient GANs via lipsCHitz continuity constrAIned Normalization

How Does Lipschitz Regularization Influence GAN Training?

Direct Adversarial Training: An Adaptive Method to Penalize Lipschitz Continuity of the Discriminator

Adversarial Lipschitz Regularization

Wasserstein Bounded Generative Adversarial Networks

Loss-Sensitive Generative Adversarial Networks on Lipschitz Densities

Complexity Controlled Generative Adversarial Networks

Which Training Methods for GANs do actually Converge?

GraN-GAN: Piecewise Gradient Normalization for Generative Adversarial Networks

Direct Adversarial Training for GANs

Understanding GANs in the LQG Setting: Formulation, Generalization and Stability

Improving Generative Adversarial Networks with Local Coordinate Coding

Stabilizing Training of Generative Adversarial Nets via Langevin Stein Variational Gradient Descent