Text Feature Adversarial Learning for Text Generation With Knowledge Transfer From GPT2
Hao Zhang,Yulai Cong,Zhengjue Wang,Lei Zhang,Miaoyun Zhao,Liqun Chen,Shijing Si,Ricardo Henao,Lawrence Carin
DOI: https://doi.org/10.1109/tnnls.2022.3210975
IF: 14.255
2022-01-01
IEEE Transactions on Neural Networks and Learning Systems
Abstract:Text generation is a key component of many natural language tasks. Motivated by the success of generative adversarial networks (GANs) for image generation, many text-specific GANs have been proposed. However, due to the discrete nature of text, these text GANs often use reinforcement learning (RL) or continuous relaxations to calculate gradients during learning, leading to high-variance or biased estimation. Furthermore, the existing text GANs often suffer from mode collapse (i.e., they have limited generative diversity). To tackle these problems, we propose a new text GAN model named text feature GAN (TFGAN), where adversarial learning is performed in a continuous text feature space. In the adversarial game, GPT2 provides the "true" features, while the generator of TFGAN learns from them. TFGAN is trained by maximum likelihood estimation on text space and adversarial learning on text feature space, effectively combining them into a single objective, while alleviating mode collapse. TFGAN achieves appealing performance in text generation tasks, and it can also be used as a flexible framework for learning text representations.
computer science, artificial intelligence, theory & methods,engineering, electrical & electronic, hardware & architecture