Abstract:While large language models (LLMs) have proven to be effective on a large variety of tasks, they are also known to hallucinate information. To measure whether an LLM prefers factually consistent continuations of its input, we propose a new benchmark called FIB(Factual Inconsistency Benchmark) that focuses on the task of summarization. Specifically, our benchmark involves comparing the scores an LLM assigns to a factually consistent versus a factually inconsistent summary for an input news article. For factually consistent summaries, we use human-written reference summaries that we manually verify as factually consistent. To generate summaries that are factually inconsistent, we generate summaries from a suite of summarization models that we have manually annotated as factually inconsistent. A model's factual consistency is then measured according to its accuracy, i.e.\ the proportion of documents where it assigns a higher score to the factually consistent summary. To validate the usefulness of FIB, we evaluate 23 large language models ranging from 1B to 176B parameters from six different model families including BLOOM and OPT. We find that existing LLMs generally assign a higher score to factually consistent summaries than to factually inconsistent summaries. However, if the factually inconsistent summaries occur verbatim in the document, then LLMs assign a higher score to these factually inconsistent summaries than factually consistent summaries. We validate design choices in our benchmark including the scoring method and source of distractor summaries. Our code and benchmark data can be found at <a class="link-external link-https" href="https://github.com/r-three/fib" rel="external noopener nofollow">this https URL</a>.

Joint Contrastive Learning for Factual Consistency Evaluation of Cross-Lingual Abstract Summarization

Evaluating Factuality in Cross-lingual Summarization

Leveraging Entailment Judgements in Cross-Lingual Summarisation

Revisiting Cross-Lingual Summarization: A Corpus-based Study and A New Benchmark with Improved Annotation

CONFIT: Toward Faithful Dialogue Summarization with Linguistically-Informed Contrastive Fine-tuning

CO2Sum:Contrastive Learning for Factual-Consistent Abstractive Summarization

ConVerSum: A Contrastive Learning based Approach for Data-Scarce Solution of Cross-Lingual Summarization Beyond Direct Equivalents

Contrastive Aligned Joint Learning for Multilingual Summarization.

Improving Factual Error Correction for Abstractive Summarization via Data Distillation and Conditional-generation Cloze

Improving Abstractive Dialogue Summarization with Speaker-Aware Supervised Contrastive Learning.

Improving Factuality of Abstractive Summarization via Contrastive Reward Learning

Abstract Summarization Model Based on Semantic Graphs and Entity Pointers

Evaluating Factual Consistency of Summaries with Large Language Models

Factual Dialogue Summarization via Learning from Large Language Models

On Improving Summarization Factual Consistency from Natural Language Feedback

Factual Consistency Evaluation of Summarisation in the Era of Large Language Models

Evaluating the Factual Consistency of Large Language Models Through News Summarization

Identifying Factual Inconsistencies in Summaries: Grounding LLM Inference via Task Taxonomy

NCLS: Neural Cross-Lingual Summarization

Understanding Translationese in Cross-Lingual Summarization

Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Models