Abstract:Maintaining factual consistency is a critical issue in abstractive text summarisation, however, it cannot be assessed by traditional automatic metrics used for evaluating text summarisation, such as ROUGE scoring. Recent efforts have been devoted to developing improved metrics for measuring factual consistency using pre-trained language models, but these metrics have restrictive token limits, and are therefore not suitable for evaluating long document text summarisation. Moreover, there is limited research and resources available for evaluating whether existing automatic evaluation metrics are fit for purpose when applied in long document settings. In this work, we evaluate the efficacy of automatic metrics for assessing the factual consistency of long document text summarisation. We create a human-annotated data set for evaluating automatic factuality metrics, LongSciVerify, which contains fine-grained factual consistency annotations for long document summaries from the scientific domain. We also propose a new evaluation framework, LongDocFACTScore, which is suitable for evaluating long document summarisation. This framework allows metrics to be efficiently extended to any length document and outperforms existing state-of-the-art metrics in its ability to correlate with human measures of factuality when used to evaluate long document summarisation data sets. We make our code and LongSciVerify data set publicly available: <a class="link-external link-https" href="https://github.com/jbshp/LongDocFACTScore" rel="external noopener nofollow">this https URL</a>.

QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

Asking and Answering Questions to Evaluate the Factual Consistency of Summaries

LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation

Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization

MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarization

Improving Model Factuality with Fine-grained Critique-based Evaluator

Evaluating Factuality in Cross-lingual Summarization

Using Similarity to Evaluate Factual Consistency in Summaries

FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation

Factual Consistency Evaluation of Summarisation in the Era of Large Language Models

FIZZ: Factual Inconsistency Detection by Zoom-in Summary and Zoom-out Document

DIFFQG: Generating Questions to Summarize Factual Changes

Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering

Improving Factual Consistency in Summarization with Compression-Based Post-Editing

Evaluating Factual Consistency of Summaries with Large Language Models

Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

FAR-ASS: Fact-aware reinforced abstractive sentence summarization

On Improving Summarization Factual Consistency from Natural Language Feedback

Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization

ISQA: Informative Factuality Feedback for Scientific Summarization