BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval

Hongjin Su,Howard Yen,Mengzhou Xia,Weijia Shi,Niklas Muennighoff,Han-yu Wang,Haisu Liu,Quan Shi,Zachary S. Siegel,Michael Tang,Ruoxi Sun,Jinsung Yoon,Sercan O. Arik,Danqi Chen,Tao Yu

2024-10-24

Abstract:Existing retrieval benchmarks primarily consist of information-seeking queries (e.g., aggregated questions from search engines) where keyword or semantic-based retrieval is usually sufficient. However, many complex real-world queries require in-depth reasoning to identify relevant documents that go beyond surface form matching. For example, finding documentation for a coding question requires understanding the logic and syntax of the functions involved. To better benchmark retrieval on such challenging queries, we introduce BRIGHT, the first text retrieval benchmark that requires intensive reasoning to retrieve relevant documents. Our dataset consists of 1,384 real-world queries spanning diverse domains, such as economics, psychology, mathematics, and coding. These queries are drawn from naturally occurring and carefully curated human data. Extensive evaluation reveals that even state-of-the-art retrieval models perform poorly on BRIGHT. The leading model on the MTEB leaderboard (Muennighoff et al., 2023), which achieves a score of 59.0 nDCG@10, produces a score of nDCG@10 of 18.3 on BRIGHT. We show that incorporating explicit reasoning about the query improves retrieval performance by up to 12.2 points. Moreover, incorporating retrieved documents from the top-performing retriever boosts question-answering performance by over 6.6 points. We believe that BRIGHT paves the way for future research on retrieval systems in more realistic and challenging settings.

Computation and Language,Artificial Intelligence,Information Retrieval

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve

This paper aims to address the limitations of existing retrieval benchmark datasets (such as BEIR, MTEB, etc.) which primarily focus on information retrieval tasks (e.g., aggregation issues in search engines). These tasks are often accomplished through keyword or semantic matching. However, many complex real-world queries require deep reasoning to find relevant documents, rather than just surface-level matching. For example, finding documents for programming problems requires understanding the logic and syntax of relevant functions.

To better evaluate the retrieval performance of such complex queries, the authors introduce BRIGHT, the first text retrieval benchmark that requires extensive reasoning to retrieve relevant documents. The BRIGHT dataset contains 1,384 real-world queries from various fields, including economics, psychology, mathematics, and programming. These queries come from naturally occurring and carefully curated human data.

Through extensive evaluation, the study finds that even the current state-of-the-art retrieval models perform poorly on BRIGHT. For instance, the leading model on the MTEB leaderboard scores 59.0 nDCG@10 on the MTEB dataset but only 18.3 nDCG@10 on BRIGHT. The research shows that explicitly reasoning about the query before retrieval can significantly improve retrieval performance, with an increase of up to 12.2 points. Additionally, incorporating documents retrieved by top retrievers into downstream models can significantly enhance question-answering performance, with an increase of up to 6.6 points.

Overall, BRIGHT paves the way for future research on retrieval systems in more realistic and challenging scenarios.