Abstract:Using LLMs (Large Language Models) in conjunction with external documents has made RAG (Retrieval-Augmented Generation) an essential technology. Numerous techniques and modules for RAG are being researched, but their performance can vary across different datasets. Finding RAG modules that perform well on specific datasets is challenging. In this paper, we propose the AutoRAG framework, which automatically identifies suitable RAG modules for a given dataset. AutoRAG explores and approximates the optimal combination of RAG modules for the dataset. Additionally, we share the results of optimizing a dataset using AutoRAG. All experimental results and data are publicly available and can be accessed through our GitHub repository <a class="link-external link-https" href="https://github.com/Marker-Inc-Korea/AutoRAG_ARAGOG_Paper" rel="external noopener nofollow">this https URL</a> .

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the challenge of selecting suitable Retrieval-Augmented Generation (RAG) modules across different datasets. Although many techniques and modules for RAG are currently being researched, their performance varies across different datasets. Finding the best RAG module for a specific dataset is a challenging task. To this end, the authors propose a framework called AutoRAG, which can automatically identify the RAG module suitable for a given dataset. AutoRAG optimizes the selection process of RAG techniques by exploring and approximating the optimal combination of RAG modules. Additionally, the paper shares the results of optimizing datasets using AutoRAG and provides public access links to all experimental results and data. ### Key Points Summary 1. **Background**: - Large Language Models (LLM) combined with external documents make RAG a key technology. - The existing RAG technology selection process is often not automated and unoptimized, limiting the potential and scalability of the technology. 2. **Problem**: - It is difficult to select the best RAG module across different datasets. - The lack of systematic automated methods leads to inefficiency and an inability to comprehensively explore RAG configurations, thus affecting performance. 3. **Solution**: - Propose the AutoRAG framework to systematically evaluate different RAG settings and optimize module selection. - Through extensive experiments, similar to traditional AutoML practices in machine learning, improve the performance and scalability of RAG systems. 4. **Method**: - **Query Expansion**: Includes techniques such as direct use of user queries, query decomposition, and Hypothetical Document Embeddings (HyDE). - **Retrieval**: Includes Vector Databases (VectorDB), BM25, and hybrid retrieval methods. - **Passage Augmentation**: Increases the number of relevant passages through secondary search using metadata. - **Passage Re-ranking**: Utilizes various re-rankers such as MonoT5, Sentence Transformer, TART, etc. - **Prompt Generation**: Combines retrieved passages, user queries, and instructions into prompts for LLM to generate answers. 5. **Experiments**: - Experiments were conducted using the ARAGOG dataset, which contains 423 research papers on AI and LLM. - Evaluation metrics include Ragas Context Precision and generation quality metrics such as ROUGE, METEOR, SemScore, and G-Eval. 6. **Results**: - Experiments validated the effectiveness of the AutoRAG framework, demonstrating its ability to select the best RAG module across different datasets. - Experimental results show that certain query expansion and retrieval techniques perform better on specific datasets, while other techniques may reduce performance. In summary, this paper addresses the challenge of selecting the best RAG module across different datasets by proposing the AutoRAG framework, thereby improving the performance and scalability of RAG technology.

AutoRAG: Automated Framework for optimization of Retrieval Augmented Generation Pipeline

RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation

Auto-RAG: Autonomous Retrieval-Augmented Generation for Large Language Models

RAG-DDR: Optimizing Retrieval-Augmented Generation Using Differentiable Data Rewards

AutoRAG-HP: Automatic Online Hyper-Parameter Tuning for Retrieval-Augmented Generation

EasyRAG: Efficient Retrieval-Augmented Generation Framework for Automated Network Operations

RAGChecker: A Fine-grained Framework for Diagnosing Retrieval-Augmented Generation

RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation

RAGGED: Towards Informed Design of Retrieval Augmented Generation Systems

R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation

LightRAG: Simple and Fast Retrieval-Augmented Generation

FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research

Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data

DRAGIN: Dynamic Retrieval Augmented Generation based on the Information Needs of Large Language Models

ActiveRAG: Autonomously Knowledge Assimilation and Accommodation through Retrieval-Augmented Agents

Toward Optimal Search and Retrieval for RAG

Developing Retrieval Augmented Generation (RAG) based LLM Systems from PDFs: An Experience Report

PipeRAG: Fast Retrieval-Augmented Generation via Algorithm-System Co-design

DRAGIN: Dynamic Retrieval Augmented Generation Based on the Real-time Information Needs of Large Language Models.

ARAGOG: Advanced RAG Output Grading

Retrieval-Augmented Generation for AI-Generated Content: A Survey