A Hybrid RAG System with Comprehensive Enhancement on Complex Reasoning

Ye Yuan,Chengwu Liu,Jingyang Yuan,Gongbo Sun,Siqi Li,Ming Zhang
2024-09-02
Abstract:Retrieval-augmented generation (RAG) is a framework enabling large language models (LLMs) to enhance their accuracy and reduce hallucinations by integrating external knowledge bases. In this paper, we introduce a hybrid RAG system enhanced through a comprehensive suite of optimizations that significantly improve retrieval quality, augment reasoning capabilities, and refine numerical computation ability. We refined the text chunks and tables in web pages, added attribute predictors to reduce hallucinations, conducted LLM Knowledge Extractor and Knowledge Graph Extractor, and finally built a reasoning strategy with all the references. We evaluated our system on the CRAG dataset through the Meta CRAG KDD Cup 2024 Competition. Both the local and online evaluations demonstrate that our system significantly enhances complex reasoning capabilities. In local evaluations, we have significantly improved accuracy and reduced error rates compared to the baseline model, achieving a notable increase in scores. In the meanwhile, we have attained outstanding results in online assessments, demonstrating the performance and generalization capabilities of the proposed system. The source code for our system is released in \url{<a class="link-external link-https" href="https://gitlab.aicrowd.com/shizueyy/crag-new" rel="external noopener nofollow">this https URL</a>}.
Computation and Language,Information Retrieval
What problem does this paper attempt to address?
The paper aims to address three main challenges faced by large language models (LLMs) in complex reasoning tasks: 1. **Lack of domain-specific expertise**: Due to the limited data encountered during pre-training, LLMs may perform poorly in specialized fields such as law and medicine. The internal knowledge stored is insufficient to comprehensively handle complex legal or medical issues. 2. **Hallucination phenomenon during generation**: LLMs may generate inaccurate or inconsistent information. For example, when answering "Which number is larger, 3.11 or 3.9?", most LLMs, including GPT-4, would incorrectly think that 3.11 is larger than 3.9. 3. **Difficulty in integrating timely information**: Since the knowledge stored within LLMs is static and cannot be updated in real-time, their application is limited in rapidly changing fields such as sports and finance. These fields typically require real-time data processing, such as querying the current price of a stock or the performance of table tennis players in the Paris 2024 Olympics. To address these issues, the paper proposes an enhanced Retrieval-Augmented Generation (RAG) system that leverages external knowledge bases to enhance the capabilities of LLMs. This system improves LLMs' ability to respond in domain-specific, factual, and timely manners by extracting relevant text documents and using them in conjunction with the query for generation. The paper also details the specific design and implementation methods of the system and demonstrates its excellent performance in the CRAG benchmark tests.