Abstract:Research on Large Language Models (LLMs) has recently witnessed an increasing interest in extending models' context size to better capture dependencies within long documents. While benchmarks have been proposed to assess long-range abilities, existing efforts primarily considered generic tasks that are not necessarily aligned with real-world applications. In contrast, our work proposes a new benchmark for long-context LLMs focused on a practical meeting assistant scenario. In this scenario, the long contexts consist of transcripts obtained by automatic speech recognition, presenting unique challenges for LLMs due to the inherent noisiness and oral nature of such data. Our benchmark, named ELITR-Bench, augments the existing ELITR corpus' transcripts with 271 manually crafted questions and their ground-truth answers. Our experiments with recent long-context LLMs on ELITR-Bench highlight a gap between open-source and proprietary models, especially when questions are asked sequentially within a conversation. We also provide a thorough analysis of our GPT-4-based evaluation method, encompassing insights from a crowdsourcing study. Our findings suggest that while GPT-4's evaluation scores are correlated with human judges', its ability to differentiate among more than three score levels may be limited.

What problem does this paper attempt to address?

### Problems the Paper Aims to Solve This paper aims to address the evaluation issues of long-context language models (LLMs) in real-world application scenarios, particularly in the context of meeting assistants. Specifically: 1. **Proposing a New Benchmark**: The authors propose a new benchmark test—ELITR-Bench, designed to evaluate the performance of long-context LLMs in meeting assistant tasks. This benchmark is based on meeting transcripts generated by automatic speech recognition (ASR), which contain noise and spoken language characteristics, thus posing unique challenges to LLMs. 2. **Focus on Real-World Applications**: Unlike existing benchmarks that mainly focus on general tasks, ELITR-Bench is dedicated to real-world application scenarios, especially the meeting assistant context. In this case, the model needs to handle meeting records that can last for several hours and be able to answer questions based on these records. 3. **Evaluation in Conversational Mode**: The paper also explores the performance differences of models in conversational mode (i.e., questions are asked sequentially) and finds significant gaps between open-source models and proprietary models. This gap is particularly evident when questions are asked consecutively in a conversation. 4. **Validation of GPT-4 Evaluation Method**: The authors also validate the effectiveness of the GPT-4-based evaluation method through a crowdsourced study, noting that while GPT-4's scores correlate highly with human judges' scores, its ability to distinguish beyond three rating levels is limited. In summary, this paper introduces the ELITR-Bench benchmark test, providing new tools for evaluating the performance of long-context language models in specific real-world application scenarios, and reveals the shortcomings of existing models in handling long documents and conversational tasks.

ELITR-Bench: A Meeting Assistant Benchmark for Long-Context Language Models

CLongEval: A Chinese Benchmark for Evaluating Long-Context Large Language Models

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

L-Eval: Instituting Standardized Evaluation for Long Context Language Models

$\infty$Bench: Extending Long Context Evaluation Beyond 100K Tokens

M4LE: A Multi-Ability Multi-Range Multi-Task Multi-Domain Long-Context Evaluation Benchmark for Large Language Models

XL$^2$Bench: A Benchmark for Extremely Long Context Understanding with Long-range Dependencies

BABILong: Testing the Limits of LLMs with Long Context Reasoning-in-a-Haystack

LooGLE: Can Long-Context Language Models Understand Long Contexts?

Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark for Evaluating Long-Context Large Language Models

Leave No Document Behind: Benchmarking Long-Context LLMs with Extended Multi-Doc QA

NeedleBench: Can LLMs Do Retrieval and Reasoning in 1 Million Context Window?

ETHIC: Evaluating Large Language Models on Long-Context Tasks with High Information Coverage

LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding

MileBench: Benchmarking MLLMs in Long Context

NLPBench: Evaluating Large Language Models on Solving NLP Problems

RULER: What's the Real Context Size of Your Long-Context Language Models?

LongGenBench: Benchmarking Long-Form Generation in Long Context LLMs

Long-context LLMs Struggle with Long In-context Learning

AIR-Bench: Benchmarking Large Audio-Language Models via Generative Comprehension

LongIns: A Challenging Long-context Instruction-based Exam for LLMs