MultiLegalPile: A 689GB Multilingual Legal Corpus

Joel Niklaus,Veton Matoshi,Matthias Stürmer,Ilias Chalkidis,Daniel E. Ho

2024-05-19

Abstract:Large, high-quality datasets are crucial for training Large Language Models (LLMs). However, so far, there are few datasets available for specialized critical domains such as law and the available ones are often only for the English language. We curate and release MultiLegalPile, a 689GB corpus in 24 languages from 17 jurisdictions. The MultiLegalPile corpus, which includes diverse legal data sources with varying licenses, allows for pretraining NLP models under fair use, with more permissive licenses for the Eurlex Resources and Legal mC4 subsets. We pretrain two RoBERTa models and one Longformer multilingually, and 24 monolingual models on each of the language-specific subsets and evaluate them on LEXTREME. Additionally, we evaluate the English and multilingual models on LexGLUE. Our multilingual models set a new SotA on LEXTREME and our English models on LexGLUE. We release the dataset, the trained models, and all of the code under the most open possible licenses.

Computation and Language,Artificial Intelligence,Machine Learning

What problem does this paper attempt to address?

The main objective of this paper is to address the lack of large-scale, high-quality training datasets in the legal domain. Currently, most large language models (LLMs) are primarily trained on English and often sourced from web texts with unclear copyright status. In the paper, the authors create and release a 689GB multilingual legal corpus called "MultiLegalPile", covering 24 languages and 17 jurisdictions. This corpus consists of diverse legal data sources with different licenses, allowing pretraining of natural language processing (NLP) models under fair use terms, with more permissive licenses for Eurlex Resources and the Legal mC4 subset.

Based on this corpus, the authors further pretrained the XLM-R model and trained two legal-oriented pretrained models, both multilingual and single-language (LEGAL XLM S), evaluated on LEXTREME and LexGLUE benchmarks. Their multilingual model achieved a new state-of-the-art level on LEXTREME, while the English model performed better than other models on LexGLUE.

Additionally, the paper emphasizes the principles of open science by releasing the dataset, training models, and code under the most open licenses, aiming to promote further research and progress in the field of legal text analysis and understanding.