Abstract:Large language models (LLMs) often generate content with unsupported or unverifiable content, known as "hallucinations." To address this, retrieval-augmented LLMs are employed to include citations in their content, grounding the content in verifiable sources. Despite such developments, manually assessing how well a citation supports the associated statement remains a major challenge. Previous studies tackle this challenge by leveraging faithfulness metrics to estimate citation support automatically. However, they limit this citation support estimation to a binary classification scenario, neglecting fine-grained citation support in practical scenarios. To investigate the effectiveness of faithfulness metrics in fine-grained scenarios, we propose a comparative evaluation framework that assesses the metric effectiveness in distinguishing citations between three-category support levels: full, partial, and no support. Our framework employs correlation analysis, classification evaluation, and retrieval evaluation to measure the alignment between metric scores and human judgments comprehensively. Our results indicate no single metric consistently excels across all evaluations, highlighting the complexity of accurately evaluating fine-grained support levels. Particularly, we find that the best-performing metrics struggle to distinguish partial support from full or no support. Based on these findings, we provide practical recommendations for developing more effective metrics.

What problem does this paper attempt to address?

### Problems the Paper Attempts to Solve This paper aims to address the issue of fine-grained support level evaluation in automated citation assessment. Specifically: 1. **Background Problem**: - Content generated by large language models (LLMs) often contains unverifiable information or information that contradicts known facts, known as "hallucinations." - To mitigate this issue, researchers have adopted retrieval-augmented LLMs, which include citations when generating content to ensure verifiability. 2. **Existing Challenges**: - Manually evaluating whether citations support related statements is a labor-intensive and time-consuming task. - Automated citation assessment methods primarily rely on faithfulness metrics, but these metrics are usually limited to binary classification scenarios, i.e., determining whether a citation supports a statement, ignoring the fine-grained support levels common in practical applications (such as fully supports, partially supports, does not support). 3. **Research Objectives**: - Propose a comparative evaluation framework to assess the effectiveness of different faithfulness metrics under fine-grained support levels. - Measure the consistency of faithfulness metrics with human judgments through three methods: correlation analysis, classification evaluation, and retrieval evaluation. - Explore the performance of different types of faithfulness metrics (similarity-based and entailment-based) under different evaluation protocols. ### Main Contributions 1. **Exploration of Fine-Grained Support Levels**: - Systematically studied the impact of three support levels (fully supports, partially supports, does not support) on faithfulness metrics, marking the first systematic exploration of this issue. 2. **Proposed Comparative Evaluation Framework**: - Designed a comprehensive evaluation framework, including correlation analysis, classification evaluation, and retrieval evaluation, to thoroughly assess the effectiveness of faithfulness metrics. 3. **Experimental Results and Recommendations**: - Experimental results show that no single faithfulness metric performs well under all evaluation protocols, indicating that these evaluation protocols are complementary and should be used together. - The best-performing metrics still struggle to distinguish between partial support and full support or no support, highlighting the complexity of automated citation assessment. - Based on these findings, the authors provide practical recommendations for improving faithfulness metrics. ### Conclusion Through systematic experiments and analysis, the paper reveals the limitations of existing faithfulness metrics in handling fine-grained support levels and provides directions and suggestions for future research.

A Comparative Analysis of Faithfulness Metrics and Humans in Citation Evaluation

Towards Fine-Grained Citation Evaluation in Generated Text: A Comparative Analysis of Faithfulness Metrics

The Probabilities Also Matter: A More Faithful Metric for Faithfulness of Free-Text Explanations in Large Language Models

FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"

FaithScore: Fine-grained Evaluations of Hallucinations in Large Vision-Language Models

On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation

Evaluation of Faithfulness Using the Longest Supported Subsequence

L-CiteEval: Do Long-Context Models Truly Leverage Context for Responding?

Evaluating Human Alignment and Model Faithfulness of LLM Rationale

On the Capacity of Citation Generation by Large Language Models

A Meta-Evaluation of Faithfulness Metrics for Long-Form Hospital-Course Summarization

ALiiCE: Evaluating Positional Fine-grained Citation Generation

Learning Fine-Grained Grounded Citations for Attributed Large Language Models

When Large Language Models Meet Citation: A Survey

LongCite: Enabling LLMs to Generate Fine-grained Citations in Long-context QA

VALOR-EVAL: Holistic Coverage and Faithfulness Evaluation of Large Vision-Language Models

Modeling citation worthiness by using attention-based bidirectional long short-term memory networks and interpretable models

Enhancing Trust in LLMs: Algorithms for Comparing and Interpreting LLMs

Learning to Generate Answers with Citations via Factual Consistency Models

Facilitating Human-LLM Collaboration through Factuality Scores and Source Attributions

LitFM: A Retrieval Augmented Structure-aware Foundation Model For Citation Graphs