A Survey of Controllable Text Generation using Transformer-based Pre-trained Language Models

Hanqing Zhang,Haolin Song,Shaoyu Li,Ming Zhou,Dawei Song

2023-08-24

Abstract:Controllable Text Generation (CTG) is emerging area in the field of natural language generation (NLG). It is regarded as crucial for the development of advanced text generation technologies that better meet the specific constraints in practical applications. In recent years, methods using large-scale pre-trained language models (PLMs), in particular the widely used transformer-based PLMs, have become a new paradigm of NLG, allowing generation of more diverse and fluent text. However, due to the limited level of interpretability of deep neural networks, the controllability of these methods need to be guaranteed. To this end, controllable text generation using transformer-based PLMs has become a rapidly growing yet challenging new research hotspot. A diverse range of approaches have emerged in the recent 3-4 years, targeting different CTG tasks that require different types of controlled constraints. In this paper, we present a systematic critical review on the common tasks, main approaches, and evaluation methods in this area. Finally, we discuss the challenges that the field is facing, and put forward various promising future directions. To the best of our knowledge, this is the first survey paper to summarize the state-of-the-art CTG techniques from the perspective of Transformer-based PLMs. We hope it can help researchers and practitioners in the related fields to quickly track the academic and technological frontier, providing them with a landscape of the area and a roadmap for future research.

Computation and Language

What problem does this paper attempt to address?

### Problems Addressed by the Paper

This paper primarily explores key issues in the field of Controllable Text Generation (CTG) and highlights the application of large-scale Pre-trained Language Models (PLMs) based on Transformers in this domain.

Specifically:

1. **Need for Controllability**:
- In Natural Language Generation (NLG) tasks, it is crucial to ensure that the generated text meets specific constraints. These constraints can include sentiment, theme, keywords, etc.
- For example, in story generation, it is necessary to control the plot and ending; in dialogue systems, it is necessary to control sentiment, speaker style, etc.

2. **Limitations of Existing Methods**:
- Although deep learning-based methods have made progress, they still face challenges in interpretability and controllability. Existing large-scale pre-trained language models (such as BERT, GPT, etc.) can generate high-quality text, but their internal mechanisms remain black-boxed, lacking explicit control capabilities.

3. **Directions for Improvement**:
- Research how to improve the interpretability and controllability of models based on PLMs, enabling them to better meet various constraints in practical applications.
- Propose a series of new methods based on Transformer PLMs and systematically review and summarize these methods.

### Summary

This paper aims to help researchers and practitioners quickly understand the latest technologies and future development directions in the field of controllable text generation by systematically reviewing and summarizing the research progress of Transformer-based PLMs in this area.