Abstract:Large Language Models (LLMs) have shown promise in highly-specialized domains, however challenges are still present in aspects of accuracy and costs. These limitations restrict the usage of existing models in domain-specific tasks. While fine-tuning pre-trained models have shown promising results, this process can be computationally expensive and require massive datasets of the specialized application in hand. In this work, we bridge that gap. We have developed Phi-2-Legal and Mistral-Legal-7B, which are language models specifically designed for legal applications. These models are based on Phi-2 and Mistral-7B-v0.1, and have gone through continued pre-training with over 500 million tokens of legal texts. Our innovative approach significantly improves capabilities in legal tasks by using Large Language Models (LLMs) to convert raw training data into reading comprehension text. Our legal LLMs have demonstrated superior performance in legal benchmarks, even outperforming models trained on much larger datasets with more resources. This work emphasizes the effectiveness of continued pre-training on domain-specific texts, while using affordable LLMs for data conversion, which gives these models domain expertise while retaining general language understanding capabilities. While this work uses the legal domain as a test case, our method can be scaled and applied to any pre-training dataset, resulting in significant improvements across different tasks. These findings underscore the potential of domain-adaptive pre-training and reading comprehension for the development of highly effective domain-specific language models.

The Large Language Model GreekLegalRoBERTa

Exploring Large Language Models for Classical Philology

GREEK-BERT: The Greeks visiting Sesame Street

Multi-granular Legal Topic Classification on Greek Legislation

Large Language Models are legal but they are not: Making the case for a powerful LegalLLM

MultiLegalPile: A 689GB Multilingual Legal Corpus

Large Language Models "Ad Referendum": How Good Are They at Machine Translation in the Legal Domain?

Large Linguistic Models: Analyzing theoretical linguistic abilities of LLMs

Meltemi: The first open Large Language Model for Greek

Large Language Models for Scholarly Ontology Generation: An Extensive Analysis in the Engineering Field

TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Text

Large Language Models for Judicial Entity Extraction: A Comparative Study

Exploring New Frontiers of Deep Learning in Legal Practice: A Case Study of Large Language Models

Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Models

SaulLM-7B: A pioneering Large Language Model for Law

Large Language Models: A Survey

One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

Large language models as tax attorneys: a case study in legal capabilities emergence

A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction

Benchmarking Large Language Models for Log Analysis, Security, and Interpretation

Enhancing Knowledge Graph Construction Using Large Language Models