Abstract:The limited context window of contemporary large language models (LLMs) remains a huge barrier to their broader application across various domains. While continual pre-training on long-context data is a straightforward and effective solution, it incurs substantial costs in terms of data acquisition and computational resources. To alleviate this issue, we propose SharedLLM, a novel approach grounded in the design philosophy of multi-grained context compression and query-aware information retrieval. SharedLLM is composed of two short-context LLMs such as LLaMA-2, termed upper model and lower model. The lower model functions as a compressor while the upper model acts as a decoder. The upper model receives compressed, multi-grained context information from the lower model and performs context-aware modeling on the running text. Information transfer between the compressor and decoder occurs only at the lowest layers to refrain from long forward paths in the lower model and redundant cross-attention modules in the upper model. Based on this architecture, we introduce a specialized tree-style data structure to efficiently encode, store and retrieve multi-grained contextual information for text chunks. This structure, combined with a search algorithm, enables rapid encoding and retrieval of relevant information from various levels of the tree based on the input query. This entire process, wherein the sender and receiver are derived from the same LLM layer, is referred to as self-injection.

Semantic Context Enrichment and Shifting Based on Multi-dimensional Association

Model Semantic Relations with Extended Attributes

Semantic Image Segmentation Based on Spatial Context Relations

An Ontology Mapping Architecture To Facilitate Semantic Integration

Approach for multi-dimensional associated heterogeneous engineering document semantic retrieval

An Enhanced Ontology Based Context Model and Fusion Mechanism.

Modeling Contexts In Collaborative Environment: A New Approach

Integration of Contextual Descriptors in Ontology Alignment for Enrichment of Semantic Correspondence

Context query and association discovery for collaborative environment

Context Modeling for Business Collaboration

A Context Model for Collaborative Environment

Multi-Context Attention for Entity Matching.

Enhance Social Context Understanding with Semantic Chunks.

An Ontology-based Context Model in Intelligent Environments

Enhancing Semantic Spaces with Event-Driven Context Interpretation

An Ontology-Based Context Model for Building Context-Aware Services

Extending Contexts with Ontologies for Multidimensional Data Quality Assessment

Informed Multi-context Entity Alignment

CASE: Context-Aware Semantic Expansion

Two are better than one: Context window extension with multi-grained self-injection

Exploiting Correlations Between Contexts and Definitions with Multiple Definition Modeling