Long Text Generation by Modeling Sentence-Level and Discourse-Level Coherence

Jian Guan,Xiaoxi Mao,Changjie Fan,Zitao Liu,Wenbiao Ding,Minlie Huang

DOI: https://doi.org/10.48550/arXiv.2105.08963

2021-05-19

Abstract:Generating long and coherent text is an important but challenging task, particularly for open-ended language generation tasks such as story generation. Despite the success in modeling intra-sentence coherence, existing generation models (e.g., BART) still struggle to maintain a coherent event sequence throughout the generated text. We conjecture that this is because of the difficulty for the decoder to capture the high-level semantics and discourse structures in the context beyond token-level co-occurrence. In this paper, we propose a long text generation model, which can represent the prefix sentences at sentence level and discourse level in the decoding process. To this end, we propose two pretraining objectives to learn the representations by predicting inter-sentence semantic similarity and distinguishing between normal and shuffled sentence orders. Extensive experiments show that our model can generate more coherent texts than state-of-the-art baselines.

Computation and Language

What problem does this paper attempt to address?

### The Problem the Paper Attempts to Solve

The paper aims to address the issue of insufficient coherence when generating long texts. Specifically, although existing generation models (such as BART) have succeeded in modeling intra-sentence coherence, these models still struggle to maintain event sequence coherence throughout the entire text when generating long texts. The authors believe this is because, during the decoding process, the models find it difficult to capture high-level semantics and discourse structure in the context, rather than just word-level co-occurrence relationships.

To tackle this challenge, the paper proposes a new long text generation model—HINT (HIgh-level representations for loNg Text generation), which can represent sentence-level and discourse-level information of prefix sentences during the decoding process. By introducing two pre-training objectives—semantic similarity prediction and sentence order discrimination, HINT can learn more meaningful high-level representations, thereby generating more coherent long texts. Experimental results show that HINT outperforms existing state-of-the-art models on multiple benchmark tasks.