Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics

Weijia Zhang,Mohammad Aliannejadi,Yifei Yuan,Jiahuan Pei,Jia-Hong Huang,Evangelos Kanoulas

2024-08-24

Abstract:Large language models (LLMs) often produce unsupported or unverifiable content, known as "hallucinations." To mitigate this, retrieval-augmented LLMs incorporate citations, grounding the content in verifiable sources. Despite such developments, manually assessing how well a citation supports the associated statement remains a major challenge. Previous studies use faithfulness metrics to estimate citation support automatically but are limited to binary classification, overlooking fine-grained citation support in practical scenarios. To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support. Our framework employs correlation analysis, classification evaluation, and retrieval evaluation to measure the alignment between metric scores and human judgments comprehensively. Our results show no single metric consistently excels across all evaluations, revealing the complexity of assessing fine-grained support. Based on the findings, we provide practical recommendations for developing more effective metrics.

Information Retrieval,Computation and Language

What problem does this paper attempt to address?

The paper attempts to address the issue of fine-grained support for citations in generated text. Specifically, large language models (LLMs) often produce unverifiable or unsupported content, known as "hallucinations," when generating content. To mitigate this issue, retrieval-augmented LLMs introduce citations to base the content on verifiable sources. However, manually evaluating the degree of support that citations provide to related statements is a major challenge. Existing research, while using fidelity metrics to automatically estimate citation support, is mostly limited to binary classification, overlooking the fine-grained citation support in real-world scenarios.

The paper proposes a comparative evaluation framework aimed at assessing the effectiveness of different fidelity metrics in distinguishing between three levels of support (fully supported, partially supported, unsupported) for citations. This framework comprehensively measures the alignment between metric scores and human judgments through three methods: correlation analysis, classification evaluation, and retrieval evaluation. The study's results indicate that no single fidelity metric excels in all evaluations, highlighting the complexity of assessing fine-grained support. Based on these findings, the paper provides practical recommendations for developing more effective metrics.