Improving Long Text Understanding with Knowledge Distilled from Summarization Model

Yan Liu,Yazheng Yang,Xiaokang Chen

2024-05-08

Abstract:Long text understanding is important yet challenging for natural language processing. A long article or document usually contains many redundant words that are not pertinent to its gist and sometimes can be regarded as noise. With recent advances of abstractive summarization, we propose our \emph{Gist Detector} to leverage the gist detection ability of a summarization model and integrate the extracted gist into downstream models to enhance their long text understanding ability. Specifically, Gist Detector first learns the gist detection knowledge distilled from a summarization model, and then produces gist-aware representations to augment downstream models. We evaluate our method on three different tasks: long document classification, distantly supervised open-domain question answering, and non-parallel text style transfer. The experimental results show that our method can significantly improve the performance of baseline models on all tasks.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the problem of long text comprehension. Specifically, long documents often contain many redundant words that are irrelevant to the core information, which can sometimes be considered as noise. Although extractive summarization techniques have made progress in recent years, existing work rarely focuses on how to measure the importance of different parts of the text. To this end, the paper proposes a method called "Gist Detector," which leverages the ability of summarization models to extract key information and integrates this key information into downstream task models to enhance their ability to handle long texts. This method learns the key information detection capability from the summarization model through a knowledge distillation mechanism and generates key information-aware representations, thereby significantly improving the performance of baseline models on multiple tasks (such as long document classification, distant supervision open-domain question answering, and non-parallel text style transfer).