Abstract:Optimizing the discriminator in Generative Adversarial Networks (GANs) to completion in the inner training loop is computationally prohibitive, and on finite datasets would result in overfitting. To address this, a common update strategy is to alternate between k optimization steps for the discriminator D and one optimization step for the generator G. This strategy is repeated in various GAN algorithms where k is selected empirically. In this paper, we show that this update strategy is not optimal in terms of accuracy and convergence speed, and propose a new update strategy for Wasserstein GANs (WGAN) and other GANs using the WGAN loss(e.g. WGAN-GP, Deblur GAN, and Super-resolution GAN). The proposed update strategy is based on a loss change ratio comparison of G and D. We demonstrate that the proposed strategy improves both convergence speed and accuracy.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is in the training process of Generative Adversarial Networks (GANs), especially Wasserstein GAN (WGAN) and its related variants (such as WGAN - GP, Deblur GAN, Super Resolution GAN), how to optimize the update strategies between the generator (G) and the discriminator (D) to improve the training efficiency and the final performance. Traditional update strategies usually adopt a fixed ratio (for example, update the discriminator five times before updating the generator once). This strategy may not be optimal in practice because it does not consider the dynamic balance of the loss change rates of the generator and the discriminator. The paper proposes a new update strategy based on the comparison of the generator and the discriminator loss change rates. Specifically, this strategy calculates the change rates of the generator and the discriminator losses and decides whether to update the generator or the discriminator next according to these change rates. This method aims to accelerate the convergence speed and improve the model accuracy by adaptively adjusting the update frequencies of the generator and the discriminator. ### Main Contributions 1. **Propose an adaptive update strategy**: This strategy is based on the comparison of the generator and the discriminator loss change rates, and dynamically adjusts the update frequencies of the generator and the discriminator instead of using a fixed update ratio. 2. **Accelerate the training process**: The experimental results show that compared with the traditional fixed - update strategy, the proposed adaptive update strategy can reach convergence faster and shows higher performance in many cases. 3. **Prove local convergence**: By analyzing the Dirac - GAN problem, it is proved that the proposed adaptive update strategy can reach a local minimum point, while the traditional fixed - update strategy cannot achieve this. ### Experimental Verification The paper conducts experiments on different GAN models on multiple datasets, including: - **WGAN** and **WGAN - GP**: For image generation tasks. - **Deblur GAN**: For image deblurring tasks. - **Super Resolution WGAN**: For image super - resolution tasks. The experimental results show that the proposed adaptive update strategy not only accelerates the training process but also improves the model performance in many cases. ### Conclusion By proposing an adaptive update strategy based on the generator and the discriminator loss change rates, the paper effectively solves the problems existing in the traditional fixed - update strategy and improves the training efficiency and the final performance of WGAN and its related variants. This method provides a new idea for GAN training and has important theoretical and practical application values.

Accelerated WGAN update strategy with loss change rate balancing

GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium

Closing the Gap Between Theory and Practice During Alternating Optimization for GANs

Incremental Focal Loss GANs.

Soft Generative Adversarial Network: Combating Mode Collapse in Generative Adversarial Network Training Via Dynamic Borderline Softening Mechanism

Two steps at a time -- taking GAN training in stride with Tseng's method

Gradient descent GAN optimization is locally stable

Optimizing Latent Distributions for Non-Adversarial Generative Networks

Adaptive Weighted Discriminator for Training Generative Adversarial Networks

Effective Lipschitz constraint enforcement for Wasserstein GAN training

GL-GAN: Adaptive Global and Local Bilevel Optimization model of Image Generation

Gang of GANs: Generative Adversarial Networks with Maximum Margin Ranking

Adaptive Spectral Normalization for Generative Models

Understanding the Effectiveness of Lipschitz Constraint in Training of GANs Via Gradient Analysis.

Improving GANs with A Dynamic Discriminator

Wasserstein GANs with Gradient Penalty Compute Congested Transport

MCGAN: Enhancing GAN Training with Regression-Based Generator Loss

Wasserstein Generative Adversarial Networks

Local Stability of Wasserstein GANs With Abstract Gradient Penalty

On the Convergence and Robustness of Training GANs with Regularized Optimal Transport