Abstract:Recent strides in Large Language Models (LLMs) have saturated many Natural Language Processing (NLP) benchmarks, emphasizing the need for more challenging ones to properly assess LLM capabilities. However, domain-specific and multilingual benchmarks are rare because they require in-depth expertise to develop. Still, most public models are trained predominantly on English corpora, while other languages remain understudied, particularly for practical domain-specific NLP tasks. In this work, we introduce a novel NLP benchmark for the legal domain that challenges LLMs in five key dimensions: processing \emph{long documents} (up to 50K tokens), using \emph{domain-specific knowledge} (embodied in legal texts), \emph{multilingual} understanding (covering five languages), \emph{multitasking} (comprising legal document-to-document Information Retrieval, Court View Generation, Leading Decision Summarization, Citation Extraction, and eight challenging Text Classification tasks) and \emph{reasoning} (comprising especially Court View Generation, but also the Text Classification tasks). Our benchmark contains diverse datasets from the Swiss legal system, allowing for a comprehensive study of the underlying non-English, inherently multilingual legal system. Despite the large size of our datasets (some with hundreds of thousands of examples), existing publicly available multilingual models struggle with most tasks, even after extensive in-domain pre-training and fine-tuning. We publish all resources (benchmark suite, pre-trained models, code) under permissive open CC BY-SA licenses.

EUROPA: A Legal Multilingual Keyphrase Generation Dataset

EuroLLM: Multilingual Language Models for Europe

A Dataset of German Legal Documents for Named Entity Recognition

Unlocking Legal Knowledge: A Multilingual Dataset for Judicial Summarization in Switzerland

Multi-EuP: The Multilingual European Parliament Dataset for Analysis of Bias in Information Retrieval

Towards Multilingual LLM Evaluation for European Languages

Retrieval-Augmented Multilingual Keyphrase Generation with Retriever-Generator Iterative Training

MultiLegalPile: A 689GB Multilingual Legal Corpus

Legal Prompt Engineering for Multilingual Legal Judgement Prediction

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

Eurolects and EU Legal Translation

One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

Teuken-7B-Base & Teuken-7B-Instruct: Towards European LLMs

Keyphrase Generation Beyond the Boundaries of Title and Abstract

LLM vs. Lawyers: Identifying a Subset of Summary Judgments in a Large UK Case Law Dataset

LeXFiles and LegalLAMA: Facilitating English Multinational Legal Language Model Development

Keyphrase Generation: A Multi-Aspect Survey

LAR-ECHR: A New Legal Argument Reasoning Task and Dataset for Cases of the European Court of Human Rights

A Survey of Large Language Models for European Languages

The JRC-Acquis: A multilingual aligned parallel corpus with 20+ languages

Building cross-language corpora for human understanding of privacy policies