IdentifyMe: A Challenging Long-Context Mention Resolution Benchmark

Kawshik Manikantan,Makarand Tapaswi,Vineet Gandhi,Shubham Toshniwal

2024-11-12

Abstract:Recent evaluations of LLMs on coreference resolution have revealed that traditional output formats and evaluation metrics do not fully capture the models' referential understanding. To address this, we introduce IdentifyMe, a new benchmark for mention resolution presented in a multiple-choice question (MCQ) format, commonly used for evaluating LLMs. IdentifyMe features long narratives and employs heuristics to exclude easily identifiable mentions, creating a more challenging task. The benchmark also consists of a curated mixture of different mention types and corresponding entities, allowing for a fine-grained analysis of model performance. We evaluate both closed- and open source LLMs on IdentifyMe and observe a significant performance gap (20-30%) between the state-of-the-art sub-10B open models vs. closed ones. We observe that pronominal mentions, which have limited surface information, are typically much harder for models to resolve than nominal mentions. Additionally, we find that LLMs often confuse entities when their mentions overlap in nested structures. The highest-scoring model, GPT-4o, achieves 81.9% accuracy, highlighting the strong referential capabilities of state-of-the-art LLMs while also indicating room for further improvement.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The core problem that this paper attempts to solve is the insufficient performance of current language models in the coreference resolution task, especially the inadequate evaluation of the entity recognition and parsing capabilities mentioned in long texts. Traditional output formats and evaluation metrics cannot fully capture these models' understanding ability of the mentioned entities. To meet this challenge, the author introduced a new benchmark named IdentifyMe. This benchmark is presented in the form of multiple - choice questions, specifically designed for evaluating large - language models, contains long - form narratives, and excludes easily recognizable mentions through heuristic methods, thereby creating a more challenging task. In addition, IdentifyMe also includes a series of different mention types and their corresponding entities, which enables a detailed analysis of model performance. Through this benchmark, the author hopes to more accurately evaluate and improve the ability of large - language models to handle complex coreference relationships.