Document-aware Positional Encoding and Linguistic-guided Encoding for Abstractive Multi-document Summarization

Congbo Ma,Wei Emma Zhang,Pitawelayalage Dasun Dileepa Pitawela,Yutong Qu,Haojie Zhuang,Hu Wang

DOI: https://doi.org/10.48550/arXiv.2209.05929

2022-09-13

Abstract:One key challenge in multi-document summarization is to capture the relations among input documents that distinguish between single document summarization (SDS) and multi-document summarization (MDS). Few existing MDS works address this issue. One effective way is to encode document positional information to assist models in capturing cross-document relations. However, existing MDS models, such as Transformer-based models, only consider token-level positional information. Moreover, these models fail to capture sentences' linguistic structure, which inevitably causes confusions in the generated summaries. Therefore, in this paper, we propose document-aware positional encoding and linguistic-guided encoding that can be fused with Transformer architecture for MDS. For document-aware positional encoding, we introduce a general protocol to guide the selection of document encoding functions. For linguistic-guided encoding, we propose to embed syntactic dependency relations into the dependency relation mask with a simple but effective non-linear encoding learner for feature learning. Extensive experiments show the proposed model can generate summaries with high quality.

Computation and Language

What problem does this paper attempt to address?

This paper attempts to solve two main problems in multi - document summarization:

1. **Capturing cross - document relationships**: In the multi - document summarization (MDS) task, the model needs to be able to capture the relationships between different documents, which is a problem that single - document summarization (SDS) does not need to handle. Existing MDS models, such as Transformer - based models, usually only consider the word - level position information and ignore the document - level position information. This deficiency makes it difficult for the model to detect cross - document relationships, thus affecting the quality of the summary.

2. **Preserving syntactic structures**: Although existing MDS models can explicitly calculate the relationships between each word through the self - attention mechanism, they lack explicit support for syntactic structures. This will lead to the generated summary content being irrelevant or deviating from the original meaning. Therefore, how to preserve the syntactic structures of the source documents when generating summaries is also an important issue.

To solve the above problems, the author proposes two encoding mechanisms:

- **Document - aware Positional Encoding**: By introducing a general protocol to guide the selection of document encoding functions and combining document - level position information with word - level position information, to help the model better capture cross - document relationships.

- **Linguistic - guided Encoding**: By embedding dependency relationships into the dependency relation mask and using a simple non - linear encoding learner for feature learning, to help the model preserve the correct dependency structures and grammatical associations when generating summaries.

The combination of these two encoding mechanisms aims to improve the quality of multi - document summarization, making the generated summaries more fluent, information - rich and grammatically correct. Experimental results show that the proposed model outperforms existing baseline models on multiple evaluation metrics.