Integrating Planning into Single-Turn Long-Form Text Generation

Yi Liang,You Wu,Honglei Zhuang,Li Chen,Jiaming Shen,Yiling Jia,Zhen Qin,Sumit Sanghai,Xuanhui Wang,Carl Yang,Michael Bendersky

2024-10-09

Abstract:Generating high-quality, in-depth textual documents, such as academic papers, news articles, Wikipedia entries, and books, remains a significant challenge for Large Language Models (LLMs). In this paper, we propose to use planning to generate long form content. To achieve our goal, we generate intermediate steps via an auxiliary task that teaches the LLM to plan, reason and structure before generating the final text. Our main novelty lies in a single auxiliary task that does not require multiple rounds of prompting or planning. To overcome the scarcity of training data for these intermediate steps, we leverage LLMs to generate synthetic intermediate writing data such as outlines, key information and summaries from existing full articles. Our experiments demonstrate on two datasets from different domains, namely the scientific news dataset SciNews and Wikipedia datasets in KILT-Wiki and FreshWiki, that LLMs fine-tuned with the auxiliary task generate higher quality documents. We observed +2.5% improvement in ROUGE-Lsum, and a strong 3.60 overall win/loss ratio via human SxS evaluation, with clear wins in organization, relevance, and verifiability.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to use large - language models (LLMs) to generate high - quality, in - depth long - text documents, such as academic papers, news articles, Wikipedia entries, and books. Although existing large - language models have made remarkable progress in various text - generation tasks, generating coherent and substantial long - form documents remains a major challenge. To this end, the paper proposes a new method to simplify the generation process and enhance the consistency of documents by directly fine - tuning LLMs in a single call to generate the entire long - form document.

Specifically, the main contributions of the paper are as follows:

1. **Proposing a new method**: This method directly fine - tunes LLMs to generate complete long - form documents in one call, simplifying the generation process and enhancing the consistency of documents.
2. **Introducing the pre - writing stage**: Inspired by human writing practices, the proposed framework teaches LLMs to plan and structure documents before generating the final text by introducing auxiliary training tasks.
3. **Generating synthetic intermediate writing data**: To overcome the challenge of limited training data in intermediate writing steps, use the capabilities of LLMs to generate synthetic summaries, outlines, and key information from existing complete articles. This innovative method provides rich training data for the learning of LLMs.
4. **Experimental results**: Extensive experimental results show that LLMs fine - tuned with auxiliary tasks generate higher - quality and more coherent long - form documents on multiple datasets.

Through these contributions, the paper aims to improve the ability of LLMs to generate high - quality long - form documents, making them more suitable for practical applications.