Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More?

Jinhyuk Lee,Anthony Chen,Zhuyun Dai,Dheeru Dua,Devendra Singh Sachan,Michael Boratko,Yi Luan,Sébastien M. R. Arnold,Vincent Perot,Siddharth Dalmia,Hexiang Hu,Xudong Lin,Panupong Pasupat,Aida Amini,Jeremy R. Cole,Sebastian Riedel,Iftekhar Naim,Ming-Wei Chang,Kelvin Guu

2024-06-19

Abstract:Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. Leveraging LCLMs' ability to natively ingest and process entire corpora of information offers numerous advantages. It enhances user-friendliness by eliminating the need for specialized knowledge of tools, provides robust end-to-end modeling that minimizes cascading errors in complex pipelines, and allows for the application of sophisticated prompting techniques across the entire system. To assess this paradigm shift, we introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks. However, LCLMs still face challenges in areas like compositional reasoning that are required in SQL-like tasks. Notably, prompting strategies significantly influence performance, emphasizing the need for continued research as context lengths grow. Overall, LOFT provides a rigorous testing ground for LCLMs, showcasing their potential to supplant existing paradigms and tackle novel tasks as model capabilities scale.

Computation and Language,Artificial Intelligence,Information Retrieval

What problem does this paper attempt to address?

The paper primarily explores the performance of Long-Context Language Models (LCLMs) across various tasks and proposes a benchmarking framework named LOFT to evaluate these models' capabilities. Specifically, the paper attempts to address the following questions:

1. **Information Retrieval**: Can LCLMs directly retrieve relevant information from a large corpus of text, thereby replacing traditional retrieval systems?
2. **Retrieval-Augmented Generation (RAG)**: Can LCLMs simplify RAG systems by directly handling the entire corpus to reduce retrieval errors?
3. **SQL Queries**: Can LCLMs handle entire databases to perform natural language queries, thus avoiding the need to convert natural language into SQL code?
4. **Many-Shot In-Context Learning (ICL)**: Can LCLMs improve performance with a large number of examples, thereby eliminating the need to carefully select a few examples?
5. **Corpus-in-Context Prompting (CiC)**: Proposes a new prompting method that enables LCLMs to better utilize long contexts for reasoning and task execution.

Through the evaluation of the aforementioned areas, the paper demonstrates that LCLMs perform comparably to specially optimized models on certain tasks but still have shortcomings in complex multi-hop reasoning tasks. Additionally, the paper emphasizes the impact of prompting strategies on model performance and suggests directions for future research.