Abstract:This paper studies stable learning methods for generative models that enable high-quality data generation. Noise injection is commonly used to stabilize learning. However, selecting a suitable noise distribution is challenging. Diffusion-GAN, a recently developed method, addresses this by using the diffusion process with a timestep-dependent discriminator. We investigate Diffusion-GAN and reveal that data scaling is a key component for stable learning and high-quality data generation. Building on our findings, we propose a learning algorithm, Scale-GAN, that uses data scaling and variance-based regularization. Furthermore, we theoretically prove that data scaling controls the bias-variance trade-off of the estimation error bound. As a theoretical extension, we consider GAN with invertible data augmentations. Comparative evaluations on benchmark datasets demonstrate the effectiveness of our method in improving stability and accuracy.

What problem does this paper attempt to address?

This paper attempts to solve the stability problems encountered during the training process of generative models (especially Generative Adversarial Networks, GANs). Specifically, the author focuses on how to achieve high - quality data generation through data scaling and variance regularization, and improve the stability of the learning process. The following are the main problems and solutions in this paper: ### 1. Stability problems Generative Adversarial Networks (GANs) often encounter the following problems during the training process: - **Lack of convergence**: Training may not converge to the optimal solution. - **Mode Collapse**: The generator can only generate a limited variety of samples and cannot cover the entire data distribution. - **Catastrophic Forgetting**: The model forgets old data when learning new data. - **Overfitting**: The model overfits the training data, resulting in a decline in generalization ability. ### 2. The role of data scaling The author studied Diffusion - GAN (a GAN combined with a diffusion model) and found that data scaling is a key factor in stable learning and high - quality data generation. Specifically: - **Small - scale scaling**: It can make the gradient change of the discriminator more gentle, reduce sudden gradient changes, and thus avoid early learning failures. - **Large - scale scaling**: Although it can learn the discriminant boundary faster, it is prone to instability, especially before generating accurate samples. ### 3. The role of variance regularization To further stabilize the learning process, the author introduced a variance - based regularization method. This method uses the variance of the discriminator as an approximation of gradient regularization, providing a simpler and more effective method to control the distribution of the generator. The specific formula is: \[ \mathbb{E}_t[V_{p_t}[eD]] \] where \( V_{p_t} \) is the variance with respect to the scaled data distribution \( p_t \), and \( eD(y, t) \) is the discriminator. ### 4. The proposed new framework: Scale - GAN Based on the above analysis, the author proposed a new learning algorithm - Scale - GAN, whose main features include: - **Data scaling**: Stabilize the learning process by scaling data. - **Variance regularization**: Use variance regularization to prevent overfitting and mode collapse. - **Multi - scale learning**: Conduct learning at multiple scales simultaneously to combine the advantages of different scales. ### 5. Theoretical analysis The author theoretically proved that data scaling does not reduce the efficiency of generator learning, and variance regularization can control the bias - variance trade - off of the estimated error bound to a certain extent. The specific formula is: \[ \mathbb{E}_{z}[\nabla_\theta G_\theta(z) \nabla_y eD(G_\theta(z), 0)-\frac{1}{1 + eD(G_\theta(z), 0)}] \] The derivation of this formula ensures the invariance of the gradient direction between the generator and the discriminator, thereby alleviating the problem of unbalanced convergence speeds between the two. ### Conclusion Through numerical experiments and theoretical analysis, the author proved the effectiveness of Scale - GAN in improving stability and accuracy, especially its performance on benchmark datasets such as image generation is better than existing methods.

Scaling-based Data Augmentation for Generative Models and its Theoretical Extension

Toward Understanding Generative Data Augmentation

Generative Expansion of Small Datasets: An Expansive Graph Approach

Differentiable Augmentation for Data-Efficient GAN Training

Augmenting data with generative adversarial networks: An overview

Diffusion-GAN: Training GANs with Diffusion

On Data Augmentation for GAN Training

DifAugGAN: A Practical Diffusion-style Data Augmentation for GAN-based Single Image Super-resolution

On Stabilizing Generative Adversarial Training with Noise

Learning from Bad Data Via Generation

Effective Data Augmentation with Multi-Domain Learning GANs

LatentAugment: Data Augmentation via Guided Manipulation of GAN's Latent Space

Stabilizing Training of Generative Adversarial Networks through Regularization

Performance Study of Image Data Augmentation by Generative Adversarial Networks

MS$^3$D: A RG Flow-Based Regularization for GAN Training with Limited Data

ScoreMix: A Scalable Augmentation Strategy for Training GANs with Limited Data

Data Augmentation for Image Classification using Generative AI

ScalingGaussian: Enhancing 3D Content Creation with Generative Gaussian Splatting

Data Augmentation Using GANs

Image Augmentations for GAN Training

MCGAN: Enhancing GAN Training with Regression-Based Generator Loss