Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

Minzheng Wang,Longze Chen,Cheng Fu,Shengyi Liao,Xinghua Zhang,Bingli Wu,Haiyang Yu,Nan Xu,Lei Zhang,Run Luo,Yunshui Li,Min Yang,Fei Huang,Yongbin Li

2024-10-03

Abstract:Long-context modeling capabilities have garnered widespread attention, leading to the emergence of Large Language Models (LLMs) with ultra-context windows. Meanwhile, benchmarks for evaluating long-context LLMs are gradually catching up. However, existing benchmarks employ irrelevant noise texts to artificially extend the length of test cases, diverging from the real-world scenarios of long-context applications. To bridge this gap, we propose a novel long-context benchmark, Loong, aligning with realistic scenarios through extended multi-document question answering (QA). Unlike typical document QA, in Loong's test cases, each document is relevant to the final answer, ignoring any document will lead to the failure of the answer. Furthermore, Loong introduces four types of tasks with a range of context lengths: Spotlight Locating, Comparison, Clustering, and Chain of Reasoning, to facilitate a more realistic and comprehensive evaluation of long-context understanding. Extensive experiments indicate that existing long-context language models still exhibit considerable potential for enhancement. Retrieval augmented generation (RAG) achieves poor performance, demonstrating that Loong can reliably assess the model's long-context modeling capabilities.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The paper aims to address the evaluation issues of large language models (LLMs) in multi-document question answering tasks in long text scenarios. Specifically:

- **Limitations of existing benchmarks**: Current benchmarks often artificially extend the length of test cases by adding irrelevant noise text, which deviates from the long text processing scenarios in real applications.
- **Proposing a new benchmark**: The paper introduces a new benchmark called Loong, which simulates real scenarios by extending multi-document question answering (QA). In Loong, each document is related to the final answer, and ignoring any document will result in a failure to answer.
- **Various task types**: Loong introduces four types of tasks—Spotlight Locating, Comparison, Clustering, and Chain of Reasoning—to comprehensively evaluate long text understanding capabilities.
- **Contexts of different lengths**: Loong covers contexts ranging from 10K to 250K in length, making the evaluation more detailed.

Overall, Loong aims to fill the gap in existing long text understanding benchmarks, ensuring that evaluations are closer to real-world application scenarios and can more accurately measure the performance of current LLMs in handling complex long text tasks.