FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

Orion Weller,Benjamin Chang,Sean MacAvaney,Kyle Lo,Arman Cohan,Benjamin Van Durme,Dawn Lawrie,Luca Soldaini

2024-05-07

Abstract:Modern Language Models (LMs) are capable of following long and complex instructions that enable a large and diverse set of user requests. While Information Retrieval (IR) models use these LMs as the backbone of their architectures, virtually none of them allow users to provide detailed instructions alongside queries, thus limiting their ability to satisfy complex information needs. In this work, we study the use of instructions in IR systems. First, we introduce our dataset FollowIR, which contains a rigorous instruction evaluation benchmark as well as a training set for helping IR models learn to better follow real-world instructions. FollowIR repurposes detailed instructions -- also known as narratives -- developed for professional assessors to evaluate retrieval systems. In particular, we build our benchmark from three collections curated for shared tasks at the Text REtrieval Conference (TREC). These collections contains hundreds to thousands of labeled documents per query, making them suitable for our exploration. Through this process, we can measure how well IR models follow instructions, through a new pairwise evaluation framework. Our results indicate that existing retrieval models fail to correctly use instructions, using them for basic keywords and struggling to understand long-form information. However, we show that it is possible for IR models to learn to follow complex instructions: our new FollowIR-7B model has significant improvements after fine-tuning on our training set.

Information Retrieval,Computation and Language,Machine Learning

What problem does this paper attempt to address?

This paper attempts to address the inadequacy of Information Retrieval (IR) models in handling complex instructions. Specifically, modern Language Models (LMs) are capable of following long and complex instructions to fulfill various user requests, but existing IR models typically do not leverage these capabilities, being limited to simple keyword matching and unable to handle detailed instructions. This limits their ability to meet complex information needs.

To tackle this challenge, the paper proposes the FOLLOW IR dataset, which includes a rigorous instruction evaluation benchmark and a diverse set of real-world instructions for training. By repurposing detailed instructions (also known as narratives) developed by professional assessors for IR systems, the paper constructs an evaluation benchmark and proposes a new pairwise evaluation framework to measure the ability of retrieval models to follow instructions. Experimental results show that existing retrieval models perform poorly in using instructions, often relying only on keywords in the instructions rather than understanding their overall meaning. However, the paper demonstrates a new fine-tuned model, FOLLOW IR-7B, which achieves significant improvements in following complex instructions. Therefore, the main contributions of the paper are providing a benchmark for evaluating instruction-following capabilities, analyzing why current models struggle to understand instructions, and offering a training dataset and open-source model to teach retrieval models to better follow instructions.