Large Language Models for Information Retrieval: A Survey

Yutao Zhu,Huaying Yuan,Shuting Wang,Jiongnan Liu,Wenhan Liu,Chenlong Deng,Haonan Chen,Zheng Liu,Zhicheng Dou,Ji-Rong Wen

2024-09-04

Abstract:As a primary means of information acquisition, information retrieval (IR) systems, such as search engines, have integrated themselves into our daily lives. These systems also serve as components of dialogue, question-answering, and recommender systems. The trajectory of IR has evolved dynamically from its origins in term-based methods to its integration with advanced neural models. While the neural models excel at capturing complex contextual signals and semantic nuances, thereby reshaping the IR landscape, they still face challenges such as data scarcity, interpretability, and the generation of contextually plausible yet potentially inaccurate responses. This evolution requires a combination of both traditional methods (such as term-based sparse retrieval methods with rapid response) and modern neural architectures (such as language models with powerful language understanding capacity). Meanwhile, the emergence of large language models (LLMs), typified by ChatGPT and GPT-4, has revolutionized natural language processing due to their remarkable language understanding, generation, generalization, and reasoning abilities. Consequently, recent research has sought to leverage LLMs to improve IR systems. Given the rapid evolution of this research trajectory, it is necessary to consolidate existing methodologies and provide nuanced insights through a comprehensive overview. In this survey, we delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers. Additionally, we explore promising directions, such as search agents, within this expanding field.

Computation and Language,Information Retrieval

What problem does this paper attempt to address?

The problem this paper attempts to address is the balance between traditional methods and modern neural models in Information Retrieval (IR) systems. Specifically, although neural models excel at capturing complex contextual signals and semantic nuances, they still face challenges such as data scarcity, poor interpretability, and the potential to generate contextually plausible but inaccurate responses. Therefore, researchers are exploring how to leverage large language models (LLMs), such as ChatGPT and GPT-4, to improve the performance of information retrieval systems.

The main objectives of the paper are:
1. **Review existing methods**: Integrate existing methods of applying LLMs to IR systems and provide a detailed overview.
2. **Explore key components**: Conduct an in-depth analysis of the application of LLMs in key components such as Query Rewriter, Retriever, Reranker, and Reader.
3. **Explore future directions**: Discuss future research directions, particularly the application of Search Agents.

Through these objectives, the paper aims to provide researchers and practitioners with a comprehensive perspective on how LLMs are reshaping the current and future landscape of the information retrieval field.