Yusen Zhang,Ansong Ni,Ziming Mao,Chen Henry Wu,Chenguang Zhu,Budhaditya Deb,Ahmed H. Awadallah,Dragomir Radev,Rui Zhang
Abstract:Text summarization helps readers capture salient information from documents, news, interviews, and meetings. However, most state-of-the-art pretrained language models (LM) are unable to efficiently process long text for many summarization tasks. In this paper, we propose Summ$^N$, a simple, flexible, and effective multi-stage framework for input texts that are longer than the maximum context length of typical pretrained LMs. Summ$^N$ first splits the data samples and generates a coarse summary in multiple stages and then produces the final fine-grained summary based on it. Our framework can process input text of arbitrary length by adjusting the number of stages while keeping the LM input size fixed. Moreover, it can deal with both single-source documents and dialogues, and it can be used on top of different backbone abstractive summarization models. To the best of our knowledge, Summ$^N$ is the first multi-stage split-then-summarize framework for long input summarization. Our experiments demonstrate that Summ$^N$ outperforms previous state-of-the-art methods by improving ROUGE scores on three long meeting summarization datasets AMI, ICSI, and QMSum, two long TV series datasets from SummScreen, and a long document summarization dataset GovReport. Our data and code are available at <a class="link-external link-https" href="https://github.com/psunlpgroup/Summ-N" rel="external noopener nofollow">this https URL</a>.
What problem does this paper attempt to address?
The problem that this paper attempts to solve is that in the long - text summarization generation task, existing pre - trained language models (LMs) are unable to efficiently process long texts due to their input length limitations. Specifically, most state - of - the - art pre - trained language models perform poorly when dealing with long - text summarization tasks that exceed their maximum context lengths. To address this issue, the paper proposes a multi - stage framework named **SUMMN**, which can handle texts longer than the input lengths of traditional pre - trained models and can process input texts of arbitrary lengths by adjusting the number of stages while keeping the model input size fixed.
### Main problems:
1. **Insufficient long - text processing ability**: When processing long texts, existing pre - trained language models are unable to effectively capture all the information in the texts due to input length limitations.
2. **Dependence on leading bias**: Some methods handle long texts by truncating the input or using a retrieval - summarization pipeline, but these methods will break the context - dependency relationship and reduce the number of tokens that the model can read.
3. **Limitations of simplified attention mechanisms**: To handle longer inputs, some methods optimize the attention mechanism in the Transformer, but this will weaken the capabilities of the pre - trained model.
### Solutions:
- **Multi - stage split - summarization framework**: SUMMN first divides the input text into multiple paragraphs, then generates rough summaries in multiple stages, and finally generates a fine - grained summary in the last stage.
- **Flexible stage adjustment**: By adjusting the number of stages, SUMMN can handle input texts of different lengths while keeping the structure of the pre - trained model intact.
- **Preserving context information**: Compared with truncating the input or using a retrieval - summarization pipeline, SUMMN only cuts at the end of each paragraph, thus preserving the context information of most sentences.
### Experimental results:
- **Performance improvement**: The experimental results show that SUMMN significantly outperforms existing methods on multiple long - text summarization datasets. In particular, on datasets such as AMI, ICSI, QMSum, SummScreen, and GovReport, SUMMN has achieved significant improvements in ROUGE scores.
- **Generalization ability for different backbone models**: SUMMN is not only applicable to the BART model, but also to other backbone models such as T5 and PEGASUS, and can significantly improve performance in all cases.
### Summary:
This paper proposes an innovative multi - stage framework, SUMMN, which effectively solves the problem of insufficient processing ability of existing models in the long - text summarization generation task. Through multi - stage splitting and summary generation, SUMMN can process input texts of arbitrary lengths while keeping the model input size fixed and performs well on multiple datasets.