ISQA: Informative Factuality Feedback for Scientific Summarization

Zekai Li,Yanxia Qin,Qian Liu,Min-Yen Kan

2024-04-20

Abstract:We propose Iterative Facuality Refining on Informative Scientific Question-Answering (ISQA) feedback\footnote{Code is available at \url{

Computation and Language

What problem does this paper attempt to address?

On Improving Summarization Factual Consistency from Natural Language Feedback

Yixin Liu,Budhaditya Deb,Milagro Teruel,Aaron Halfaker,Dragomir Radev,Ahmed H. Awadallah

DOI: https://doi.org/10.48550/arXiv.2212.09968

2023-10-16

Abstract:Despite the recent progress in language generation models, their outputs may not always meet user expectations. In this work, we study whether informational feedback in natural language can be leveraged to improve generation quality and user preference alignment. To this end, we consider factual consistency in summarization, the quality that the summary should only contain information supported by the input documents, as the user-expected preference. We collect a high-quality dataset, DeFacto, containing human demonstrations and informational natural language feedback consisting of corrective instructions, edited summaries, and explanations with respect to the factual consistency of the summary. Using our dataset, we study three natural language generation tasks: (1) editing a summary by following the human feedback, (2) generating human feedback for editing the original summary, and (3) revising the initial summary to correct factual errors by generating both the human feedback and edited summary. We show that DeFacto can provide factually consistent human-edited summaries and further insights into summarization factual consistency thanks to its informational natural language feedback. We further demonstrate that fine-tuned language models can leverage our dataset to improve the summary factual consistency, while large language models lack the zero-shot learning ability in our proposed tasks that require controllable text generation.

Computation and Language
FAR-ASS: Fact-aware reinforced abstractive sentence summarization

Mengli Zhang,Gang Zhou,Wanting Yu,Wenfen Liu

DOI: https://doi.org/10.1016/j.ipm.2020.102478

2021-05-01

Abstract:<p>Automatic summarization systems provide an effective solution to today's unprecedented growth of textual data. For real-world tasks, such as data mining and information retrieval, the factual correctness of generated summary is critical. However, existing models usually focus on improving the informativeness rather than optimizing factual correctness. In this work, we present a Fact-Aware Reinforced ive Sentence Summarization framework to improve the factual correctness of neural abstractive summarization models, denoted as FAR-ASS. Specifically, we develop an automatic fact extraction scheme leveraging OpenIE (Open Information Extraction) and dependency parser tools to extract structured fact tuples. Then, to quantitatively evaluate the factual correctness, we define a factual correctness score function that considers the factual accuracy and factual redundancy. We further propose to adopt reinforcement learning to improve readability and factual correctness by jointly optimizing a mixed-objective learning function. We use the English Gigaword and DUC 2004 datasets to evaluate our model. Experimental results show that compared with competitive models, our model significantly improves the factual correctness and readability of generated summaries, and also reduces duplicates while improving the informativeness.</p>

computer science, information systems,information science & library science
Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

Alex Wang,Kyunghyun Cho,Mike Lewis

DOI: https://doi.org/10.48550/arXiv.2004.04228

2020-04-09

Abstract:Practical applications of abstractive summarization models are limited by frequent factual inconsistencies with respect to their input. Existing automatic evaluation metrics for summarization are largely insensitive to such errors. We propose an automatic evaluation protocol called QAGS (pronounced "kags") that is designed to identify factual inconsistencies in a generated summary. QAGS is based on the intuition that if we ask questions about a summary and its source, we will receive similar answers if the summary is factually consistent with the source. To evaluate QAGS, we collect human judgments of factual consistency on model-generated summaries for the CNN/DailyMail (Hermann et al., 2015) and XSUM (Narayan et al., 2018) summarization datasets. QAGS has substantially higher correlations with these judgments than other automatic evaluation metrics. Also, QAGS offers a natural form of interpretability: The answers and questions generated while computing QAGS indicate which tokens of a summary are inconsistent and why. We believe QAGS is a promising tool in automatically generating usable and factually consistent text.

Computation and Language
Improving Factuality of Abstractive Summarization via Contrastive Reward Learning

I-Chun Chern,Zhiruo Wang,Sanjan Das,Bhavuk Sharma,Pengfei Liu,Graham Neubig

2023-07-10

Abstract:Modern abstractive summarization models often generate summaries that contain hallucinated or contradictory information. In this paper, we propose a simple but effective contrastive learning framework that incorporates recent developments in reward learning and factuality metrics. Empirical studies demonstrate that the proposed framework enables summarization models to learn from feedback of factuality metrics using contrastive reward learning, leading to more factual summaries by human evaluations. This suggests that further advances in learning and evaluation algorithms can feed directly into providing more factual summaries.

Computation and Language,Artificial Intelligence
Improving Query-Focused Meeting Summarization with Query-Relevant Knowledge

Tiezheng Yu,Ziwei Ji,Pascale Fung

DOI: https://doi.org/10.48550/arXiv.2309.02105

2023-09-05

Abstract:Query-Focused Meeting Summarization (QFMS) aims to generate a summary of a given meeting transcript conditioned upon a query. The main challenges for QFMS are the long input text length and sparse query-relevant information in the meeting transcript. In this paper, we propose a knowledge-enhanced two-stage framework called Knowledge-Aware Summarizer (KAS) to tackle the challenges. In the first stage, we introduce knowledge-aware scores to improve the query-relevant segment extraction. In the second stage, we incorporate query-relevant knowledge in the summary generation. Experimental results on the QMSum dataset show that our approach achieves state-of-the-art performance. Further analysis proves the competency of our methods in generating relevant and faithful summaries.

Computation and Language,Artificial Intelligence
Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization

Yue Zhang,Jingxuan Zuo,Liqiang Jing

2024-10-22

Abstract:Multimodal summarization aims to generate a concise summary based on the input text and image. However, the existing methods potentially suffer from unfactual output. To evaluate the factuality of multimodal summarization models, we propose two fine-grained and explainable evaluation frameworks (FALLACIOUS) for different application scenarios, i.e. reference-based factuality evaluation framework and reference-free factuality evaluation framework. Notably, the reference-free factuality evaluation framework doesn't need ground truth and hence it has a wider application scenario. To evaluate the effectiveness of the proposed frameworks, we compute the correlation between our frameworks and the other metrics. The experimental results show the effectiveness of our proposed method. We will release our code and dataset via github.

Computation and Language
Evaluating Factuality in Cross-lingual Summarization

Mingqi Gao,Wenqing Wang,Xiaojun Wan,Yuemei Xu

2023-01-01

Abstract:Cross-lingual summarization aims to help people efficiently grasp the core idea of the document written in a foreign language. Modern text summarization models generate highly fluent but often factually inconsistent outputs, which has received heightened attention in recent research. However, the factual consistency of cross-lingual summarization has not been investigated yet. In this paper, we propose a cross-lingual factuality dataset by collecting human annotations of reference summaries as well as generated summaries from models at both summary level and sentence level. Furthermore, we perform the fine-grained analysis and observe that over 50% of generated summaries and over 27% of reference summaries contain factual errors with characteristics different from monolingual summarization. Existing evaluation metrics for monolingual summarization require translation to evaluate the factuality of cross-lingual summarization and perform differently at different tasks and levels. Finally, we adapt the monolingual factuality metrics as an initial step towards the automatic evaluation of summarization factuality in cross-lingual settings. Our dataset and code are available at https://github.com/kite99520/Fact_CLS.
Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization

Yiran Chen,Pengfei Liu,Xipeng Qiu

DOI: https://doi.org/10.18653/v1/2021.findings-emnlp.179

2021-01-01

Abstract:With the continuous upgrading of the summarization systems driven by deep neural networks, researchers have higher requirements on the quality of the generated summaries, which should be not only fluent and informative but also factually correct. As a result, the field of factual evaluation has developed rapidly recently. Despite its initial progress in evaluating generated summaries, the meta-evaluation methodologies of factuality metrics are limited in their opacity, leading to the insufficient understanding of factuality metrics' relative advantages and their applicability. In this paper, we present an adversarial meta-evaluation methodology that allows us to (i) diagnose the fine-grained strengths and weaknesses of 6 existing top-performing metrics over 24 diagnostic test datasets, (ii) search for directions for further improvement by data augmentation. Our observations from this work motivate us to propose several calls for future research. We make all codes, diagnostic test datasets, trained factuality models available: https://github.com/zide05/AdvFact.
FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document

Joonho Yang,Seunghyun Yoon,Byeongjeong Kim,Hwanhee Lee

2024-10-02

Abstract:Through the advent of pre-trained language models, there have been notable advancements in abstractive summarization systems. Simultaneously, a considerable number of novel methods for evaluating factual consistency in abstractive summarization systems has been developed. But these evaluation approaches incorporate substantial limitations, especially on refinement and interpretability. In this work, we propose highly effective and interpretable factual inconsistency detection method metric Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document for abstractive summarization systems that is based on fine-grained atomic facts decomposition. Moreover, we align atomic facts decomposed from the summary with the source document through adaptive granularity expansion. These atomic facts represent a more fine-grained unit of information, facilitating detailed understanding and interpretability of the summary's factual inconsistency. Experimental results demonstrate that our proposed factual consistency checking system significantly outperforms existing systems.

Computation and Language
Dialogue Inspectional Summarization with Factual Inconsistency Awareness

Leilei Gan,Yating Zhang,Kun Kuang,Lin Yuan,Shuo Li,Changlong Sun,Xiaozhong Liu,Fei Wu

DOI: https://doi.org/10.48550/arXiv.2111.03284

2021-11-05

Abstract:Dialogue summarization has been extensively studied and applied, where the prior works mainly focused on exploring superior model structures to align the input dialogue and the output summary. However, for professional dialogues (e.g., legal debate and medical diagnosis), semantic/statistical alignment can hardly fill the logical/factual gap between input dialogue discourse and summary output with external knowledge. In this paper, we mainly investigate the factual inconsistency problem for Dialogue Inspectional Summarization (DIS) under non-pretraining and pretraining settings. An innovative end-to-end dialogue summary generation framework is proposed with two auxiliary tasks: Expectant Factual Aspect Regularization (EFAR) and Missing Factual Entity Discrimination (MFED). Comprehensive experiments demonstrate that the proposed model can generate a more readable summary with accurate coverage of factual aspects as well as informing the user with potential missing facts detected from the input dialogue for further human intervention.

Computation and Language,Artificial Intelligence
Faithful to the Original: Fact Aware Neural Abstractive Summarization

Ziqiang Cao,Furu Wei,Wenjie Li,Sujian Li

DOI: https://doi.org/10.48550/arxiv.1711.04434

2017-01-01

Abstract: Unlike extractive summarization, abstractive summarization has to fuse different parts of the source text, which inclines to create fake facts. Our preliminary study reveals nearly 30% of the outputs from a state-of-the-art neural summarization system suffer from this problem. While previous abstractive summarization approaches usually focus on the improvement of informativeness, we argue that faithfulness is also a vital prerequisite for a practical abstractive summarization system. To avoid generating fake facts in a summary, we leverage open information extraction and dependency parse technologies to extract actual fact descriptions from the source text. The dual-attention sequence-to-sequence framework is then proposed to force the generation conditioned on both the source text and the extracted fact descriptions. Experiments on the Gigaword benchmark dataset demonstrate that our model can greatly reduce fake summaries by 80%. Notably, the fact descriptions also bring significant improvement on informativeness since they often condense the meaning of the source text.
MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarization

Potsawee Manakul,Adian Liusie,Mark J. F. Gales

2023-09-08

Abstract:State-of-the-art summarization systems can generate highly fluent summaries. These summaries, however, may contain factual inconsistencies and/or information not present in the source. Hence, an important component of assessing the quality of summaries is to determine whether there is information consistency between the source and the summary. Existing approaches are typically based on lexical matching or representation-based methods. In this work, we introduce an alternative scheme based on standard information-theoretic measures in which the information present in the source and summary is directly compared. We propose a Multiple-choice Question Answering and Generation framework, MQAG, which approximates the information consistency by computing the expected statistical distance between summary and source answer distributions over automatically generated multiple-choice questions. This approach exploits multiple-choice answer probabilities, as predicted answer distributions can be compared. We conduct experiments on four summary evaluation datasets: QAG-CNNDM/XSum, XSum-Hallucination, Podcast Assessment, and SummEval. Experiments show that MQAG, using models trained on SQuAD or RACE, outperforms existing evaluation methods on the majority of tasks.

Computation and Language
Improving Factual Error Correction for Abstractive Summarization via Data Distillation and Conditional-generation Cloze

Yiyang Li,Lei Li,Dingxin Hu,Xueyi Hao,Marina Litvak,Natalia Vanetik,Yanquan Zhou

2024-02-14

Abstract:Improving factual consistency in abstractive summarization has been a focus of current research. One promising approach is the post-editing method. However, previous works have yet to make sufficient use of factual factors in summaries and suffers from the negative effect of the training datasets. In this paper, we first propose a novel factual error correction model FactCloze based on a conditional-generation cloze task. FactCloze can construct the causality among factual factors while being able to determine whether the blank can be answered or not. Then, we propose a data distillation method to generate a more faithful summarization dataset SummDSC via multiple-dimensional evaluation. We experimentally validate the effectiveness of our approach, which leads to an improvement in multiple factual consistency metrics compared to baselines.

Computer Science
Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark

Xiuying Chen,Tairan Wang,Qingqing Zhu,Taicheng Guo,Shen Gao,Zhiyong Lu,Xin Gao,Xiangliang Zhang

2024-02-22

Abstract:The summarization capabilities of pretrained and large language models (LLMs) have been widely validated in general areas, but their use in scientific corpus, which involves complex sentences and specialized knowledge, has been less assessed. This paper presents conceptual and experimental analyses of scientific summarization, highlighting the inadequacies of traditional evaluation methods, such as $n$-gram, embedding comparison, and QA, particularly in providing explanations, grasping scientific concepts, or identifying key content. Subsequently, we introduce the Facet-aware Metric (FM), employing LLMs for advanced semantic matching to evaluate summaries based on different aspects. This facet-aware approach offers a thorough evaluation of abstracts by decomposing the evaluation task into simpler subtasks.Recognizing the absence of an evaluation benchmark in this domain, we curate a Facet-based scientific summarization Dataset (FD) with facet-level annotations. Our findings confirm that FM offers a more logical approach to evaluating scientific summaries. In addition, fine-tuned smaller models can compete with LLMs in scientific contexts, while LLMs have limitations in learning from in-context information in scientific domains. This suggests an area for future enhancement of LLMs.

Computation and Language
Using Similarity to Evaluate Factual Consistency in Summaries

Yuxuan Ye,Edwin Simpson,Raul Santos Rodriguez

2024-09-23

Abstract:Cutting-edge abstractive summarisers generate fluent summaries, but the factuality of the generated text is not guaranteed. Early summary factuality evaluation metrics are usually based on n-gram overlap and embedding similarity, but are reported fail to align with human annotations. Therefore, many techniques for detecting factual inconsistencies build pipelines around natural language inference (NLI) or question-answering (QA) models with additional supervised learning steps. In this paper, we revisit similarity-based metrics, showing that this failure stems from the comparison text selection and its granularity. We propose a new zero-shot factuality evaluation metric, Sentence-BERT Score (SBERTScore), which compares sentences between the summary and the source document. It outperforms widely-used word-word metrics including BERTScore and can compete with existing NLI and QA-based factuality metrics on the benchmark without needing any fine-tuning. Our experiments indicate that each technique has different strengths, with SBERTScore particularly effective in identifying correct summaries. We demonstrate how a combination of techniques is more effective in detecting various types of error.

Computation and Language
DIFFQG: Generating Questions to Summarize Factual Changes

Jeremy R. Cole,Palak Jain,Julian Martin Eisenschlos,Michael J.Q. Zhang,Eunsol Choi,Bhuwan Dhingra

2023-03-01

Abstract:Identifying the difference between two versions of the same article is useful to update knowledge bases and to understand how articles evolve. Paired texts occur naturally in diverse situations: reporters write similar news stories and maintainers of authoritative websites must keep their information up to date. We propose representing factual changes between paired documents as question-answer pairs, where the answer to the same question differs between two versions. We find that question-answer pairs can flexibly and concisely capture the updated contents. Provided with paired documents, annotators identify questions that are answered by one passage but answered differently or cannot be answered by the other. We release DIFFQG which consists of 759 QA pairs and 1153 examples of paired passages with no factual change. These questions are intended to be both unambiguous and information-seeking and involve complex edits, pushing beyond the capabilities of current question generation and factual change detection systems. Our dataset summarizes the changes between two versions of the document as questions and answers, studying automatic update summarization in a novel way.

Computation and Language
Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering

Sungho Ko,Hyunjin Cho,Hyungjoo Chae,Jinyoung Yeo,Dongha Lee

2024-10-09

Abstract:Recent studies have investigated utilizing Knowledge Graphs (KGs) to enhance Quesetion Answering (QA) performance of Large Language Models (LLMs), yet structured KG verbalization remains challengin. Existing methods, such as triple-form or free-form textual conversion of triple-form facts, encounter several issues. These include reduced evidence density due to duplicated entities or relationships, and reduced evidence clarity due to an inability to emphasize crucial evidence. To address these issues, we propose EFSum, an Evidence-focused Fact Summarization framework for enhanced QA with knowledge-augmented LLMs. We optimize an open-source LLM as a fact summarizer through distillation and preference alignment. Our extensive experiments show that EFSum improves LLM's zero-shot QA performance, and it is possible to ensure both the helpfulness and faithfulness of the summary.

Computation and Language,Artificial Intelligence,Machine Learning
FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

Alessandro Scirè,Karim Ghonim,Roberto Navigli

2024-08-31

Abstract:Recent advancements in text summarization, particularly with the advent of Large Language Models (LLMs), have shown remarkable performance. However, a notable challenge persists as a substantial number of automatically-generated summaries exhibit factual inconsistencies, such as hallucinations. In response to this issue, various approaches for the evaluation of consistency for summarization have emerged. Yet, these newly-introduced metrics face several limitations, including lack of interpretability, focus on short document summaries (e.g., news articles), and computational impracticality, especially for LLM-based metrics. To address these shortcomings, we propose Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction (FENICE), a more interpretable and efficient factuality-oriented metric. FENICE leverages an NLI-based alignment between information in the source document and a set of atomic facts, referred to as claims, extracted from the summary. Our metric sets a new state of the art on AGGREFACT, the de-facto benchmark for factuality evaluation. Moreover, we extend our evaluation to a more challenging setting by conducting a human annotation process of long-form summarization. In the hope of fostering research in summarization factuality evaluation, we release the code of our metric and our factuality annotations of long-form summarization at <a class="link-external link-https" href="https://github.com/Babelscape/FENICE" rel="external noopener nofollow">this https URL</a>.

Computation and Language
Annotating and Modeling Fine-grained Factuality in Summarization

Tanya Goyal,Greg Durrett

DOI: https://doi.org/10.48550/arXiv.2104.04302

2021-04-09

Abstract:Recent pre-trained abstractive summarization systems have started to achieve credible performance, but a major barrier to their use in practice is their propensity to output summaries that are not faithful to the input and that contain factual errors. While a number of annotated datasets and statistical models for assessing factuality have been explored, there is no clear picture of what errors are most important to target or where current techniques are succeeding and failing. We explore both synthetic and human-labeled data sources for training models to identify factual errors in summarization, and study factuality at the word-, dependency-, and sentence-level. Our observations are threefold. First, exhibited factual errors differ significantly across datasets, and commonly-used training sets of simple synthetic errors do not reflect errors made on abstractive datasets like XSum. Second, human-labeled data with fine-grained annotations provides a more effective training signal than sentence-level annotations or synthetic data. Finally, we show that our best factuality detection model enables training of more factual XSum summarization models by allowing us to identify non-factual tokens in the training data.

Computation and Language

ISQA: Informative Factuality Feedback for Scientific Summarization

On Improving Summarization Factual Consistency from Natural Language Feedback

FAR-ASS: Fact-aware reinforced abstractive sentence summarization

Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

Improving Factuality of Abstractive Summarization via Contrastive Reward Learning

Improving Query-Focused Meeting Summarization with Query-Relevant Knowledge

Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization

Evaluating Factuality in Cross-lingual Summarization

Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization

FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document

Dialogue Inspectional Summarization with Factual Inconsistency Awareness

Faithful to the Original: Fact Aware Neural Abstractive Summarization

MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarization

Improving Factual Error Correction for Abstractive Summarization via Data Distillation and Conditional-generation Cloze

Rethinking Scientific Summarization Evaluation: Grounding Explainable Metrics on Facet-aware Benchmark

Using Similarity to Evaluate Factual Consistency in Summaries

DIFFQG: Generating Questions to Summarize Factual Changes

Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering

FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

Annotating and Modeling Fine-grained Factuality in Summarization