Prompt-Based Length Controlled Generation with Multiple Control Types

Renlong Jie,Xiaojun Meng,Lifeng Shang,Xin Jiang,Qun Liu

2024-06-12

Abstract:Large language models (LLMs) have attracted great attention given their strong performance on a wide range of NLP tasks. In practice, users often expect generated texts to fall within a specific length range, making length controlled generation an important topic, especially for GPT-style models. Existing length control methods mostly focus on a simple control type of "equal to" a target length. Different from them, we propose a prompt-based method to achieve length controlled generation under different control types with high accuracy. In particular, we adopt reinforcement learning (RL) and sample filtering with the reward signal given by rule-based reward models, which enhances the length control ability of models by rewarding outputs that follow certain control instructions. In addition, we introduce a standard prompt extractor to parse arbitrary users' input into standard control instructions. Experiments show that our method significantly improves the accuracy of prompt-based length control on popular summarization datasets like CNNDM and NYT under multiple control types. Moreover, both the standard prompt extractor and RL-tuned model show strong generalization to unseen control prompt templates.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the issue of length control in text generation by large language models (such as GPT-style models). Specifically, existing length control methods mostly focus on the simple control type of making the generated text length equal to a target length. However, in practical applications, users may require various types of length control, such as generating text with a length greater than, less than, or within a certain range.

The paper proposes a prompt-based method that can generate text with high accuracy under different types of length control. To achieve this goal, the authors employ reinforcement learning (specifically the Proximal Policy Optimization algorithm, PPO) and a rule-based reward model to enhance the model's length control capability. Additionally, a Standard Prompt Extractor (SPE) is introduced to parse user input and convert it into standardized control instructions. Experimental results show that this method significantly improves length control accuracy under multiple control types and demonstrates strong generalization ability on common summarization datasets (such as CNNDM and NYT). Overall, this research aims to develop a method capable of handling various length control requirements to better meet the needs of practical application scenarios.