Abstract:Generative Adversarial Networks (GANs) have been studied in text generation to tackle the exposure bias problem. Despite their remarkable development, they adopt autoregressive structures so suffering from high latency in both training and inference stages. Although GANs have potential to support efficient generation by adopting non-autoregressive (NAR) structures, their explorations in NAR models are extremely limited. In this work, we conduct pioneering study of building language GANs based on NAR structures. We identify two issues that constrain the performance of GAN-based NAR models. Firstly, existing methods of incorporating latent variables provide highly similar representations which cannot describe the diversity of different words in sentences. We tackle this problem by proposing Position-Aware Self-Modulation, providing more diverse and effective representations. Secondly, the attention mechanism in Transformer cannot accurately build word dependencies in the unstable training of GANs, and we adopt Dependency Feed Forward Network to enhance the model capacity in dependency modeling. Armed with these two facilities, we propose a GAN-based NAR model, Adversarial Non-autoregressive Transformer (ANT). The experimental results demonstrate that ANT can achieve comparable performance with mainstream models in a single forward pass and has great potential in various applications like latent interpolation and semi-supervised learning.

Adversarial Rap Lyric Generation

Creative and Diverse Artwork Generation Using Adversarial Networks

Raply: A profanity-mitigated rap generator

DopeLearning: A Computational Approach to Rap Lyrics Generation

DeepRapper: Neural Rap Generation with Rhyme and Rhythm Modeling

Creative GANs for generating poems, lyrics, and metaphors

Conditional LSTM-GAN for Melody Generation from Lyrics

Evaluating Creative Language Generation: The Case of Rap Lyric Ghostwriting

Generating Natural Language Adversarial Examples on a Large Scale with Generative Models

R&R: Metric-guided Adversarial Sentence Generation

ARAML: A Stable Adversarial Training Framework for Text Generation

ReGAN: RE[LAX|BAR|INFORCE] based Sequence Generation using GANs

Drop the beat! Freestyler for Accompaniment Conditioned Rapping Voice Generation

A General Framework for Learning Prosodic-Enhanced Representation of Rap Lyrics

A Text GAN for Language Generation with Non-Autoregressive Generator

Unlocking the Power of GANs in Non-Autoregressive Text Generation

Adversarial Sub-sequence for Text Generation.

SongNet: Rigid Formats Controlled Text Generation

Adversarial Feature Matching for Text Generation

Text Generation Based on Generative Adversarial Nets with Latent Variable

R-GAN: Exploring Human-like Way for Reasonable Text-to-Image Synthesis via Generative Adversarial Networks