Multi-Reranker: Maximizing performance of retrieval-augmented generation in the FinanceRAG challenge

Joohyun Lee,Minji Roh
2024-11-23
Abstract:As Large Language Models (LLMs) increasingly address domain-specific problems, their application in the financial sector has expanded rapidly. Tasks that are both highly valuable and time-consuming, such as analyzing financial statements, disclosures, and related documents, are now being effectively tackled using LLMs. This paper details the development of a high-performance, finance-specific Retrieval-Augmented Generation (RAG) system for the ACM-ICAIF '24 FinanceRAG competition. We optimized performance through ablation studies on query expansion and corpus refinement during the pre-retrieval phase. To enhance retrieval accuracy, we employed multiple reranker models. Notably, we introduced an efficient method for managing long context sizes during the generation phase, significantly improving response quality without sacrificing performance. We ultimately achieve 2nd place in the FinanceRAG Challenge. Our key contributions include: (1) pre-retrieval ablation analysis, (2) an enhanced retrieval algorithm, and (3) a novel approach for long-context management. This work demonstrates the potential of LLMs in effectively processing and analyzing complex financial data to generate accurate and valuable insights. The source code and further details are available at <a class="link-external link-https" href="https://github.com/cv-lee/FinanceRAG" rel="external noopener nofollow">this https URL</a>.
Computation and Language
What problem does this paper attempt to address?
This paper aims to solve the problem of using large - language models (LLMs) for efficient information retrieval and generation in the financial field. Specifically, the author has developed a high - performance, finance - domain - specific Retrieval - Augmented Generation (RAG) system to meet the challenges in the ACM - ICAIF '24 FinanceRAG competition. The main objectives of the paper include: 1. **Optimizing pre - retrieval techniques**: Optimize query expansion and corpus optimization techniques through ablation studies to improve the overall retrieval efficiency. 2. **Improving retrieval accuracy**: Utilize multiple reranker models to enhance retrieval accuracy, thereby obtaining relevant corpora more precisely. 3. **Managing long - context**: Design an effective method to handle long - context in the generation phase, ensuring high performance and high - quality responses when dealing with a large amount of input data. Through these methods, the author has successfully improved the performance of the system and achieved the second place in the FinanceRAG competition. The key contributions of the paper are: - **Pre - retrieval ablation study**: Conducted a comprehensive ablation study, optimized pre - retrieval techniques, and improved the overall retrieval effect. - **Accurate retrieval algorithm**: Developed an accurate retrieval algorithm based on multiple reranker models, which improved the relevance of retrieved corpora. - **Efficient context management**: Designed an effective method to manage long - context, which can handle very long input data without sacrificing performance. These contributions demonstrate the ability of large - language models to process and analyze complex financial data, providing new ideas and methods for the application of AI technology in the financial industry.