Abstract:Large Language Models (LLMs) have shown promise in highly-specialized domains, however challenges are still present in aspects of accuracy and costs. These limitations restrict the usage of existing models in domain-specific tasks. While fine-tuning pre-trained models have shown promising results, this process can be computationally expensive and require massive datasets of the specialized application in hand. In this work, we bridge that gap. We have developed Phi-2-Legal and Mistral-Legal-7B, which are language models specifically designed for legal applications. These models are based on Phi-2 and Mistral-7B-v0.1, and have gone through continued pre-training with over 500 million tokens of legal texts. Our innovative approach significantly improves capabilities in legal tasks by using Large Language Models (LLMs) to convert raw training data into reading comprehension text. Our legal LLMs have demonstrated superior performance in legal benchmarks, even outperforming models trained on much larger datasets with more resources. This work emphasizes the effectiveness of continued pre-training on domain-specific texts, while using affordable LLMs for data conversion, which gives these models domain expertise while retaining general language understanding capabilities. While this work uses the legal domain as a test case, our method can be scaled and applied to any pre-training dataset, resulting in significant improvements across different tasks. These findings underscore the potential of domain-adaptive pre-training and reading comprehension for the development of highly effective domain-specific language models.

Getting in the Door: Streamlining Intake in Civil Legal Services with Large Language Models

Intention and Context Elicitation with Large Language Models in the Legal Aid Intake Process

Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models

Analyzing Images of Legal Documents: Toward Multi-Modal LLMs for Access to Justice

Large Language Models are legal but they are not: Making the case for a powerful LegalLLM

Equitable Access to Justice: Logical LLMs Show Promise

Large language models for automated Q&A involving legal documents: a survey on algorithms, frameworks and applications

Large Language Models and Explainable Law: a Hybrid Methodology

InternLM-Law: An Open Source Chinese Legal Large Language Model

Leveraging Large Language Models for Learning Complex Legal Concepts through Storytelling

Evaluation Ethics of LLMs in Legal Domain

LawLLM: Law Large Language Model for the US Legal System

Exploring the Nexus of Large Language Models and Legal Systems: A Short Survey

Legal Evalutions and Challenges of Large Language Models

A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction

TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Text

Alignment Between the Decision-Making Logic of LLMs and Human Cognition: A Case Study on Legal LLMs

Large language models as tax attorneys: a case study in legal capabilities emergence

One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

Exploring New Frontiers of Deep Learning in Legal Practice: A Case Study of Large Language Models

Enhancing Logical Reasoning in Large Language Models to Facilitate Legal Applications