Abstract:Modern LLMs can now produce highly readable abstractive summaries, to the point where traditional automated metrics for evaluating summary quality, such as ROUGE, have become saturated. However, LLMs still sometimes introduce unwanted content into summaries, i.e., information inconsistent with or unsupported by their source. Measuring the occurrence of these often subtle ``hallucinations'' automatically has proved to be challenging. This in turn has motivated development of a variety of metrics intended to measure the factual consistency of generated summaries against their source. But are these approaches measuring what they purport to do? In this work, we stress-test automatic factuality metrics. Specifically, we investigate whether and to what degree superficial attributes of summary texts suffice to predict ``factuality'', finding that a (supervised) model using only such shallow features is reasonably competitive with SOTA factuality scoring methods. We then evaluate how factuality metrics respond to factual corrections in inconsistent summaries and find that only a few show meaningful improvements. In contrast, some metrics are more sensitive to benign, non-factual edits. Motivated by these insights, we show that one can ``game'' (most) automatic factuality metrics, i.e., reliably inflate ``factuality'' scores by appending innocuous sentences to generated <a class="link-external link-http" href="http://summaries.Taken" rel="external noopener nofollow">this http URL</a> together, our results raise questions about the degree to which we should rely on existing automated factuality metrics and what exactly we want ``factuality metrics'' to measure.

Are Factuality Checkers Reliable? Adversarial Meta-evaluation of Factuality in Summarization

Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation

Evaluating Factuality in Cross-lingual Summarization

Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics

Evaluating the Tradeoff Between Abstractiveness and Factuality in Abstractive Summarization

LongDocFACTScore: Evaluating the Factuality of Long Document Abstractive Summarisation

Evaluating Factuality in Text Simplification

Annotating and Modeling Fine-grained Factuality in Summarization

Fine-grained and Explainable Factuality Evaluation for Multimodal Summarization

Improving Model Factuality with Fine-grained Critique-based Evaluator

QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarization

FENICE: Factuality Evaluation of summarization based on Natural language Inference and Claim Extraction

Questioning the Validity of Summarization Datasets and Improving Their Factual Consistency

AMRFact: Enhancing Summarization Factuality Evaluation with AMR-Driven Negative Samples Generation

Models See Hallucinations: Evaluating the Factuality in Video Captioning

Using Similarity to Evaluate Factual Consistency in Summaries

The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey

Improving Factuality of Abstractive Summarization via Contrastive Reward Learning

A Critical Look at Meta-evaluating Summarisation Evaluation Metrics

MetaSumPerceiver: Multimodal Multi-Document Evidence Summarization for Fact-Checking