RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

Dongyu Ru,Lin Qiu,Xiangkun Hu,Tianhang Zhang,Peng Shi,Shuaichen Chang,Cheng Jiayang,Cunxiang Wang,Shichao Sun,Huanyu Li,Zizhao Zhang,Binjie Wang,Jiarong Jiang,Tong He,Zhiguo Wang,Pengfei Liu,Yue Zhang,Zheng Zhang
2024-08-17
Abstract:Despite Retrieval-Augmented Generation (RAG) showing promising capability in leveraging external knowledge, a comprehensive evaluation of RAG systems is still challenging due to the modular nature of RAG, evaluation of long-form responses and reliability of measurements. In this paper, we propose a fine-grained evaluation framework, RAGChecker, that incorporates a suite of diagnostic metrics for both the retrieval and generation modules. Meta evaluation verifies that RAGChecker has significantly better correlations with human judgments than other evaluation metrics. Using RAGChecker, we evaluate 8 RAG systems and conduct an in-depth analysis of their performance, revealing insightful patterns and trade-offs in the design choices of RAG architectures. The metrics of RAGChecker can guide researchers and practitioners in developing more effective RAG systems. This work has been open sourced at <a class="link-external link-https" href="https://github.com/amazon-science/RAGChecker" rel="external noopener nofollow">this https URL</a>.
Computation and Language,Artificial Intelligence
What problem does this paper attempt to address?
The paper aims to address several key challenges in the evaluation of Retrieval-Augmented Generation (RAG) systems. Specifically: 1. **Module Complexity**: Since RAG systems consist of both a retriever and a generator module, designing effective evaluation metrics becomes complex. It is necessary to establish metrics that can comprehensively evaluate the entire system and the interactions between its modules. 2. **Metric Limitations**: Existing evaluation metrics for RAG systems are often rule-based or coarse-grained, failing to provide accurate and interpretable results. Traditional retriever metrics like recall@k and MRR rely on annotated snippets and rigid segmentation methods, while commonly used generator metrics such as n-gram (e.g., BLEU, ROUGE), embedding-based (e.g., BERTScore), or large language model (LLM) methods perform poorly when handling long answers. 3. **Metric Reliability**: The effectiveness of existing metrics has not been fully validated. Effective evaluation metrics should not only accurately reflect system performance but also align with human judgment to ensure their practicality in real-world scenarios. To address these challenges, the paper proposes a new evaluation framework called RAGCHECKER, which is based on fine-grained claim-level entailment checking. This framework allows for detailed analysis of the retrieval and generation processes and provides a set of diagnostic metrics. Through extensive experiments, the paper demonstrates that RAGCHECKER outperforms other evaluation metrics on multiple benchmark datasets and reveals some important patterns and trade-offs in RAG system design.