Abstract:Semantic textual similartiy (STS) and information retrieval tasks (IR) tasks have been the two major avenues to record the progress of embedding models in the past few years. Under the emerging Retrieval-augmented Generation (RAG) paradigm, we envision the need to evaluate next-level language understanding abilities of embedding models, and take a conscious look at the reasoning abilities stored in them. Addressing this, we pose the question: Can retrievers solve reasoning problems? By transforming reasoning tasks into retrieval tasks, we find that without specifically trained for reasoning-level language understanding, current state-of-the-art retriever models may still be far from being competent for playing the role of assisting LLMs, especially in reasoning-intensive tasks. Moreover, albeit trained to be aware of instructions, instruction-aware IR models are often better off without instructions in inference time for reasoning tasks, posing an overlooked retriever-LLM behavioral gap for the research community to align. However, recent decoder-based embedding models show great promise in narrowing the gap, highlighting the pathway for embedding models to achieve reasoning-level language understanding. We also show that, although current off-the-shelf re-ranker models fail on these tasks, injecting reasoning abilities into them through fine-tuning still appears easier than doing so to bi-encoders, and we are able to achieve state-of-the-art performance across all tasks by fine-tuning a reranking model. We release Reasoning as Retrieval Benchmark (RAR-b), a holistic suite of tasks and settings to evaluate the reasoning abilities stored in retriever models. RAR-b is available at

Task-aware Retrieval with Instructions

ControlRetriever: Harnessing the Power of Instructions for Controllable Retrieval

INSTRUCTIR: A Benchmark for Instruction Following of Information Retrieval Models

FollowIR: Evaluating and Teaching Information Retrieval Models to Follow Instructions

I3: Intent-Introspective Retrieval Conditioned on Instructions

Retrieval Augmented Instruction Tuning for Open NER with Large Language Models

InstructUIE: Multi-task Instruction Tuning for Unified Information Extraction.

Retrieve Anyone: A General-purpose Person Re-identification Task with Instructions

INTERS: Unlocking the Power of Large Language Models in Search with Instruction Tuning

MAIR: A Massive Benchmark for Evaluating Instructed Retrieval

Beyond Content Relevance: Evaluating Instruction Following in Retrieval Models

RAR-b: Reasoning as Retrieval Benchmark

Tutorial on task-based search and assistance

Unified Active Retrieval for Retrieval Augmented Generation

Active Instruction Tuning: Improving Cross-Task Generalization by Training on Prompt Sensitive Tasks

Prompt-Guided Retrieval Augmentation for Non-Knowledge-Intensive Tasks

RAG-Instruct: Boosting LLMs with Diverse Retrieval-Augmented Instructions

Improving Multitask Retrieval by Promoting Task Specialization

Visual Instruction Tuning towards General-Purpose Multimodal Model: A Survey

Distilling Instruction-following Abilities of Large Language Models with Task-aware Curriculum Planning

A Systematic Survey on Instructional Text: From Representation Formats to Downstream NLP Tasks