Abstract:Generative Adversarial Networks (GANs) have shown compelling results in various tasks and applications in recent years. However, mode collapse remains a critical problem in GANs. In this paper, we propose a novel training pipeline to address the mode collapse issue of GANs. Different from existing methods, we propose to generalize the discriminator as feature embedding and maximize the entropy of distributions in the embedding space learned by the discriminator. Specifically, two regularization terms, i.e., Deep Local Linear Embedding (DLLE) and Deep Isometric feature Mapping (DIsoMap), are designed to encourage the discriminator to learn the structural information embedded in the data, such that the embedding space learned by the discriminator can be well-formed. Based on the well-learned embedding space supported by the discriminator, a non-parametric entropy estimator is designed to efficiently maximize the entropy of embedding vectors, playing as an approximation of maximizing the entropy of the generated distribution. By improving the discriminator and maximizing the distance of the most similar samples in the embedding space, our pipeline effectively reduces the mode collapse without sacrificing the quality of generated samples. Extensive experimental results show the effectiveness of our method, which outperforms the GAN baseline, MaF-GAN on CelebA (9.13 vs. 12.43 in FID) and surpasses the recent state-of-the-art energy-based model on the ANIME-FACE dataset (2.80 vs. 2.26 in Inception score). The code is available at https://github.com/HaozheLiu-ST/MEE

Representation Degeneration Problem in Training Natural Language Generation Models

Training natural language generation mod- els

Revisiting Representation Degeneration Problem in Language Modeling.

Understanding Neural Networks through Representation Erasure.

Neural Collapse Anchored Prompt Tuning for Generalizable Vision-Language Models

Learning to Diversify Neural Text Generation via Degenerative Model

The Curious Case of Neural Text Degeneration

Language Modeling with Generative Adversarial Networks

The Stable Entropy Hypothesis and Entropy-Aware Decoding: An Analysis and Algorithm for Robust Natural Language Generation

Reconsidering Degeneration of Token Embeddings with Definitions for Encoder-based Pre-trained Language Models

Improving Diversity of Neural Text Generation Via Inverse Probability Weighting

Improving Variational Autoencoders with Density Gap-based Regularization

Tailoring Language Generation Models under Total Variation Distance

Repetition In Repetition Out: Towards Understanding Neural Text Degeneration from the Data Perspective

Improving Adversarial Neural Machine Translation for Morphologically Rich Language

Momentum Decoding: Open-ended Text Generation As Graph Exploration

Strong Model Collapse

Residual Connections Harm Generative Representation Learning

Complexity Matters: Rethinking the Latent Space for Generative Modeling

Fighting Redundancy and Model Decay with Embeddings

Combating Mode Collapse in GANs via Manifold Entropy Estimation