Abstract:In recent years, large language models (LLMs) have made remarkable achievements in various domains. However, the untimeliness and cost of knowledge updates coupled with hallucination issues of LLMs have curtailed their applications in knowledge intensive tasks, where retrieval augmented generation (RAG) can be of help. Nevertheless, existing retrieval augmented models typically use similarity as a bridge between queries and documents and follow a retrieve then read procedure. In this work, we argue that similarity is not always the panacea and totally relying on similarity would sometimes degrade the performance of retrieval augmented generation. To this end, we propose MetRag, a Multi layEred Thoughts enhanced Retrieval Augmented Generation framework. To begin with, beyond existing similarity oriented thought, we embrace a small scale utility model that draws supervision from an LLM for utility oriented thought and further come up with a smarter model by comprehensively combining the similarity and utility oriented thoughts. Furthermore, given the fact that the retrieved document set tends to be huge and using them in isolation makes it difficult to capture the commonalities and characteristics among them, we propose to make an LLM as a task adaptive summarizer to endow retrieval augmented generation with compactness-oriented thought. Finally, with multi layered thoughts from the precedent stages, an LLM is called for knowledge augmented generation. Extensive experiments on knowledge-intensive tasks have demonstrated the superiority of MetRag.

What problem does this paper attempt to address?

### The Problems Addressed by This Paper This paper primarily addresses the following issues: 1. **Existing Retrieval-Augmented Generation (RAG) methods rely on similarity metrics**: Most current RAG methods rely solely on the similarity between documents and queries to retrieve relevant information. This single similarity metric can sometimes lead to performance degradation. For example, highly similar documents are not necessarily the most useful sources of information. 2. **Information redundancy and loss**: When a large number of documents are retrieved, directly passing all documents as context to a large language model (LLM) can lead to information redundancy and loss. LLMs find it difficult to capture the commonalities and characteristics between documents when processing a large number of documents, thereby affecting performance. 3. **Outdated knowledge**: Large language models have a lag in knowledge updates, resulting in poor performance in tasks requiring the latest knowledge. Retrieval-augmented methods can compensate for this shortcoming, but existing methods are based solely on similarity retrieval and fail to fully utilize the actual utility of documents. To address these issues, the paper proposes a new framework called METRAG (Multi-layer Ered Thoughts enhanced Retrieval-Augmented Generation). This framework improves retrieval-augmented generation by combining similarity and utility thinking and introducing a task-adaptive summarization model. Specifically: - **Utility model**: Supervise the utility of documents using LLM, training a utility model that considers not only the similarity between documents and queries but also the actual utility of the documents. - **Multi-layer thinking**: Combine similarity and utility to form a more "intelligent" model that can better filter out useful documents. - **Task-adaptive summarization**: To reduce the number of documents, introduce a task-adaptive summarization model that generates compact and highly relevant summaries, further enhancing performance. With these improvements, METRAG demonstrates superior performance on multiple knowledge-intensive tasks.

Similarity is Not All You Need: Endowing Retrieval Augmented Generation with Multi Layered Thoughts

Retrieval-Augmented Generation for Large Language Models: A Survey

Meta Knowledge for Retrieval Augmented Large Language Models

Simple is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation

Retrieving, Rethinking and Revising: The Chain-of-Verification Can Improve Retrieval Augmented Generation

Corrective Retrieval Augmented Generation

IM-RAG: Multi-Round Retrieval-Augmented Generation Through Learning Inner Monologues

RAGLAB: A Modular and Research-Oriented Unified Framework for Retrieval-Augmented Generation

Metacognitive Retrieval-Augmented Large Language Models

LLMs Know What They Need: Leveraging a Missing Information Guided Framework to Empower Retrieval-Augmented Generation

MemoRAG: Moving towards Next-Gen RAG Via Memory-Inspired Knowledge Discovery

Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy

R^2AG: Incorporating Retrieval Information into Retrieval Augmented Generation

A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Models

Speculative RAG: Enhancing Retrieval Augmented Generation through Drafting

Retrieval-Augmented Generation for Natural Language Processing: A Survey

Towards Multi-Source Retrieval-Augmented Generation via Synergizing Reasoning and Preference-Driven Retrieval

MLLM Is a Strong Reranker: Advancing Multimodal Retrieval-augmented Generation via Knowledge-enhanced Reranking and Noise-injected Training

Think-then-Act: A Dual-Angle Evaluated Retrieval-Augmented Generation

WeKnow-RAG: An Adaptive Approach for Retrieval-Augmented Generation Integrating Web Search and Knowledge Graphs