Abstract:Improvements in language models' capabilities have pushed their applications towards longer contexts, making long-context evaluation and development an active research area. However, many disparate use-cases are grouped together under the umbrella term of "long-context", defined simply by the total length of the model's input, including - for example - Needle-in-a-Haystack tasks, book summarization, and information aggregation. Given their varied difficulty, in this position paper we argue that conflating different tasks by their context length is unproductive. As a community, we require a more precise vocabulary to understand what makes long-context tasks similar or different. We propose to unpack the taxonomy of long-context based on the properties that make them more difficult with longer contexts. We propose two orthogonal axes of difficulty: (I) Diffusion: How hard is it to find the necessary information in the context? (II) Scope: How much necessary information is there to find? We survey the literature on long-context, provide justification for this taxonomy as an informative descriptor, and situate the literature with respect to it. We conclude that the most difficult and interesting settings, whose necessary information is very long and highly diffused within the input, is severely under-explored. By using a descriptive vocabulary and discussing the relevant properties of difficulty in long-context, we can implement more informed research in this area. We call for a careful design of tasks and benchmarks with distinctly long context, taking into account the characteristics that make it qualitatively different from shorter context.

Exploiting Context to Identify Lexical Atoms -- A Statistical View of Linguistic Context

Semantic Context Enrichment and Shifting Based on Multi-dimensional Association

Enhance Social Context Understanding with Semantic Chunks.

A Context-Aware Approach for the Identification of Complex Words in Natural Language Texts

An Overview of Context Capturing Techniques in NLP

Shades of meaning: Uncovering the geometry of ambiguous word representations through contextualised language models

Is It Really Long Context if All You Need Is Retrieval? Towards Genuinely Difficult Long Context NLP

LSTC: Boosting Atomic Action Detection with Long-Short-Term Context.

Entity disambiguation with context awareness in user-generated short texts

Context Dependent Semantic Parsing: A Survey

Measuring Context-Word Biases in Lexical Semantic Datasets

Identifying missing dictionary entries with frequency-conserving context models

A Method for Studying Semantic Construal in Grammatical Constructions with Interpretable Contextual Embedding Spaces

Evaluating Contextualized Representations of (Spanish) Ambiguous Words: A New Lexical Resource and Empirical Analysis

Enhancing Semantic Spaces with Event-Driven Context Interpretation

Contextual Kernel and Spectral Methods for Learning the Semantics of Images

Word Sense Disambiguation Based on Positional Weighted Context

Tracing Semantic Variation in Slang

Developments in Sheaf-Theoretic Models of Natural Language Ambiguities

Capturing Paradigmatic and Syntagmatic Lexical Relations: Towards Accurate Chinese Part-of-Speech Tagging.

What Kinds of Tokens Benefit from Distant Text? An Analysis on Long Context Language Modeling