Abstract:Although large language models (LLMs) demonstrate impressive performance for many language tasks, most of them can only handle texts a few thousand tokens long, limiting their applications on longer sequence inputs, such as books, reports, and codebases. Recent works have proposed methods to improve LLMs' long context capabilities by extending context windows and more sophisticated memory mechanisms. However, comprehensive benchmarks tailored for evaluating long context understanding are lacking. In this paper, we introduce LongBench, the first bilingual, multi-task benchmark for long context understanding, enabling a more rigorous evaluation of long context understanding. LongBench comprises 21 datasets across 6 task categories in both English and Chinese, with an average length of 6,711 words (English) and 13,386 characters (Chinese). These tasks cover key long-text application areas including single-doc QA, multi-doc QA, summarization, few-shot learning, synthetic tasks, and code completion. All datasets in LongBench are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Upon comprehensive evaluation of 8 LLMs on LongBench, we find that: (1) Commercial model (GPT-3.5-Turbo-16k) outperforms other open-sourced models, but still struggles on longer contexts. (2) Scaled position embedding and fine-tuning on longer sequences lead to substantial improvement on long context understanding. (3) Context compression technique such as retrieval brings improvement for model with weak ability on long contexts, but the performance still lags behind models that have strong long context understanding capability. The code and datasets are available at <a class="link-external link-https" href="https://github.com/THUDM/LongBench" rel="external noopener nofollow">this https URL</a>.

BCWS: Bilingual Contextual Word Similarity

COS960: A Chinese Word Similarity Dataset of 960 Word Pairs.

Overview Of The Nlpcc-Iccpol 2016 Shared Task: Chinese Word Similarity Measurement

XL-WiC: A Multilingual Benchmark for Evaluating Semantic Contextualization

Constructing and validating word similarity datasets by integrating methods from psychology, brain science and computational linguistics

A Multidisciplinary Method for Constructing and Validating Word Similarity

BiCWS: Mining Cognitive Differences from Bilingual Web Search Results

SemEval-2012 Task 4: Evaluating Chinese Word Similarity.

English-Chinese Cross Language Word Embedding Similarity Calculation

Improved Word Similarity Computation for Chinese Using Sub-word Information

A Comparison of Word Embeddings for English and Cross-Lingual Chinese Word Sense Disambiguation

An English-Chinese Cross-lingual Word Semantic Similarity Measure Exploring Attributes and Relations.

Measuring Chinese-English Cross-Lingual Word Similarity With Hownet And Parallel Corpus

Wordnet Based Comparison Of Language Variation: A Study Based On Ccd And Cwn

A Dataset for Evaluating Contextualized Representation of Biomedical Concepts in Language Models

A Novel Bilingual Word Embedding Method for Lexical Translation Using Bilingual Sense Clique

Similarity Analysis of Contextual Word Representation Models

Chinese Word Similarity Computing Based on Combination Strategy

LongBench: A Bilingual, Multitask Benchmark for Long Context Understanding

Improve Statistical Machine Translation with Context-Sensitive Bilingual Semantic Embedding Model.

Cross-Language Similar Document Retrieval