Abstract:The problem of factual incorrectness in machine-generated abstractive summarization has received widespread attention in the past few years. Although large-scale neural models show excellent capability in generating fluent and coherent summaries, they still struggle with factual inconsistency, in which the named entity incorrectness is the most frequent and notable one, especially for the character-based languages, such as Chinese. Since abstractive summaries are mostly generated character by character in Chinese, the problem of hallucinated entities is more severe than that of other word-based languages. In this paper, we propose CC-Gens, a novel approach for Correctness Checking based on a Generative negative sampling strategy. Considering that the problem is due to the uncertain nature of the language generation process, we leverage fine-tuned generative language models, i.e., UniLMv2 and mT5, to generate summaries with incorrect entities, thereby constructing a synthetic binary classification dataset for the factuality discriminative model. We propose three strategies: Entity Sampling, Sequence Sampling, and Cloze Sampling, to generate summaries with incorrect entities. With such strategies, the negative samples are much more similar in nature to the output of the neural summarization model. We then train a BERT-based discriminator to identify factually incorrect machine-generated summaries with entity errors based on these generative negative samples. We further propose a novel PU learning algorithm to improve the performance of our approach by iteratively training the discriminator to select high confident negative samples from the unlabeled model generated summaries to replace the former artificially constructed ones. By generating multiple candidate summaries and selecting the one with the highest factual correctness score among them, our approach can significantly reduce the probability that an output summary contains factual errors. According to a comprehensive evaluation, the CC-Gens we proposed outperforms previous works in identifying faithless summaries as well as providing faithful ones.

Abstract Summarization Model Based on Semantic Graphs and Entity Pointers

Query-focused Multi-document Summarization: Combining a Novel Topic Model with Graph-based Semi-supervised Learning

Boundary-Aware Abstractive Summarization with Entity-Augmented Attention for Enhancing Faithfulness

Improving named entity correctness of abstractive summarization by generative negative sampling

SemSUM: Semantic Dependency Guided Neural Abstractive Summarization

Improving Factual Consistency in Summarization with Compression-Based Post-Editing

Reducing Hallucinations in Entity Abstract Summarization with Facts-Template Decomposition

Neural Entity Summarization with Joint Encoding and Weak Supervision

An entity-guided text summarization framework with relational heterogeneous graph neural network

An Abstractive Summarizer Based on Improved Pointer-Generator Network

An Entity-Driven Framework for Abstractive Summarization

Abstractive summarization incorporating graph knowledge

Faithful to the Original: Fact Aware Neural Abstractive Summarization

Leveraging Graph to Improve Abstractive Multi-Document Summarization.

GATSum: Graph-Based Topic-Aware Abstract Text Summarization

Text Summarization Generation Based on Semantic Similarity

Abstractive Document Summarization with a Graph-Based Attentional Neural Model.

A Unified Model for Extractive and Abstractive Summarization using Inconsistency Loss

Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection

Knowledge Graph-Augmented Abstractive Summarization with Semantic-Driven Cloze Reward