Abstract:Large language models (LLMs) have demonstrated strong capabilities in various aspects. However, when applying them to the highly specialized, safe-critical legal domain, it is unclear how much legal knowledge they possess and whether they can reliably perform legal-related tasks. To address this gap, we propose a comprehensive evaluation benchmark LawBench. LawBench has been meticulously crafted to have precise assessment of the LLMs' legal capabilities from three cognitive levels: (1) Legal knowledge memorization: whether LLMs can memorize needed legal concepts, articles and facts; (2) Legal knowledge understanding: whether LLMs can comprehend entities, events and relationships within legal text; (3) Legal knowledge applying: whether LLMs can properly utilize their legal knowledge and make necessary reasoning steps to solve realistic legal tasks. LawBench contains 20 diverse tasks covering 5 task types: single-label classification (SLC), multi-label classification (MLC), regression, extraction and generation. We perform extensive evaluations of 51 LLMs on LawBench, including 20 multilingual LLMs, 22 Chinese-oriented LLMs and 9 legal specific LLMs. The results show that GPT-4 remains the best-performing LLM in the legal domain, surpassing the others by a significant margin. While fine-tuning LLMs on legal specific text brings certain improvements, we are still a long way from obtaining usable and reliable LLMs in legal tasks. All data, model predictions and evaluation code are released in <a class="link-external link-https" href="https://github.com/open-compass/LawBench/" rel="external noopener nofollow">this https URL</a>. We hope this benchmark provides in-depth understanding of the LLMs' domain-specified capabilities and speed up the development of LLMs in the legal domain.

BLT: Can Large Language Models Handle Basic Legal Text?

Large Language Models are legal but they are not: Making the case for a powerful LegalLLM

LawBench: Benchmarking Legal Knowledge of Large Language Models

The unreasonable effectiveness of large language models in zero-shot semantic annotation of legal texts

A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction

Lawma: The Power of Specialization for Legal Tasks

One Law, Many Languages: Benchmarking Multilingual Legal Reasoning for Judicial Support

Caveat Lector: Large Language Models in Legal Practice

Better Call GPT, Comparing Large Language Models Against Lawyers

Fine-tuning and Application of Large Language Model in Law Domain

Large language models in cryptocurrency securities cases: can a GPT model meaningfully assist lawyers?

Large language models as tax attorneys: a case study in legal capabilities emergence

Exploring New Frontiers of Deep Learning in Legal Practice: A Case Study of Large Language Models

LexGLUE: A Benchmark Dataset for Legal Language Understanding in English

LegalBench: A Collaboratively Built Benchmark for Measuring Legal Reasoning in Large Language Models

Developing a Pragmatic Benchmark for Assessing Korean Legal Language Understanding in Large Language Models

Unlocking Practical Applications in Legal Domain: Evaluation of GPT for Zero-Shot Semantic Annotation of Legal Texts

InternLM-Law: An Open Source Chinese Legal Large Language Model

LexEval: A Comprehensive Chinese Legal Benchmark for Evaluating Large Language Models

TransformLLM: Adapting Large Language Models via LLM-Transformed Reading Comprehension Text

Exploring the psychology of LLMs' Moral and Legal Reasoning