INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning

Yutao Zhu,Peitian Zhang,Chenghao Zhang,Yifei Chen,Binyu Xie,Zheng Liu,Ji-Rong Wen,Zhicheng Dou
2024-05-28
Abstract:Large language models (LLMs) have demonstrated impressive capabilities in various natural language processing tasks. Despite this, their application to information retrieval (IR) tasks is still challenging due to the infrequent occurrence of many IR-specific concepts in natural language. While prompt-based methods can provide task descriptions to LLMs, they often fall short in facilitating a comprehensive understanding and execution of IR tasks, thereby limiting LLMs' applicability. To address this gap, in this work, we explore the potential of instruction tuning to enhance LLMs' proficiency in IR tasks. We introduce a novel instruction tuning dataset, INTERS, encompassing 20 tasks across three fundamental IR categories: query understanding, document understanding, and query-document relationship understanding. The data are derived from 43 distinct datasets with manually written templates. Our empirical results reveal that INTERS significantly boosts the performance of various publicly available LLMs, such as LLaMA, Mistral, and Phi, in IR tasks. Furthermore, we conduct extensive experiments to analyze the effects of instruction design, template diversity, few-shot demonstrations, and the volume of instructions on performance. We make our dataset and the fine-tuned models publicly accessible at <a class="link-external link-https" href="https://github.com/DaoD/INTERS" rel="external noopener nofollow">this https URL</a>.
Computation and Language,Information Retrieval
What problem does this paper attempt to address?
The problem that this paper attempts to solve is the application challenges of large - language models (LLMs) in information retrieval (IR) tasks. Although LLMs perform excellently in natural language processing (NLP) tasks, in IR tasks, the application of LLMs is limited because many IR - specific concepts have a low frequency of occurrence in natural language, and prompt - based methods are difficult to promote a comprehensive understanding and execution of IR tasks. To bridge this gap, the paper explores the potential of improving the performance of LLMs in IR tasks through instruction tuning. Specifically, the authors introduce a new instruction - tuning dataset INTERS, which aims to cover three basic IR categories: query understanding, document understanding, and query - document relationship understanding. It contains a total of 20 tasks, with data from 43 different datasets and uses manually written templates. The experimental results show that INTERS significantly improves the performance of several publicly available LLMs in IR tasks, and the effects of instruction design, template diversity, few - shot example demonstrations, and the number of instructions on performance are analyzed through extensive experiments. In addition, the paper also explores the generalization ability of the model under different settings, including generalization at the category level, task level, and dataset level.