RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation

Xuanwang Zhang,Yunze Song,Yidong Wang,Shuyun Tang,Xinfeng Li,Zhengran Zeng,Zhen Wu,Wei Ye,Wenyuan Xu,Yue Zhang,Xinyu Dai,Shikun Zhang,Qingsong Wen

2024-08-21

Abstract:Large Language Models (LLMs) demonstrate human-level capabilities in dialogue, reasoning, and knowledge retention. However, even the most advanced LLMs face challenges such as hallucinations and real-time updating of their knowledge. Current research addresses this bottleneck by equipping LLMs with external knowledge, a technique known as Retrieval Augmented Generation (RAG). However, two key issues constrained the development of RAG. First, there is a growing lack of comprehensive and fair comparisons between novel RAG algorithms. Second, open-source tools such as LlamaIndex and LangChain employ high-level abstractions, which results in a lack of transparency and limits the ability to develop novel algorithms and evaluation metrics. To close this gap, we introduce RAGLAB, a modular and research-oriented open-source library. RAGLAB reproduces 6 existing algorithms and provides a comprehensive ecosystem for investigating RAG algorithms. Leveraging RAGLAB, we conduct a fair comparison of 6 RAG algorithms across 10 benchmarks. With RAGLAB, researchers can efficiently compare the performance of various algorithms and develop novel algorithms.

Computation and Language

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address two main issues in the current research on Retrieval-Augmented Generation (RAG) algorithms:

1. **Lack of Comprehensive and Fair Comparisons**:
- In current research, many novel RAG algorithms are inconsistent in their basic components and evaluation methods, making it difficult for researchers to accurately assess the improvements of these algorithms. This poses challenges for researchers and engineers in selecting algorithms suitable for different scenarios.

2. **Insufficient Transparency of Open-Source Tools**:
- Existing open-source tools like LlamaIndex and LangChain use high-level abstractions, leading to a lack of transparency in their internal operational mechanisms. This limits the ability to develop new algorithms and evaluation metrics. Researchers need to spend a significant amount of time developing new algorithms from scratch.

To bridge these gaps, the authors introduce RAGLAB, a modular and research-oriented open-source library. RAGLAB reproduces 6 existing RAG algorithms and provides a comprehensive ecosystem for studying RAG algorithms. Through RAGLAB, researchers can efficiently compare the performance of various algorithms and develop new ones.