Chatlaw: A Multi-Agent Collaborative Legal Assistant with Knowledge Graph Enhanced Mixture-of-Experts Large Language Model

Jiaxi Cui,Munan Ning,Zongjian Li,Bohua Chen,Yang Yan,Hao Li,Bin Ling,Yonghong Tian,Li Yuan

2024-05-30

Abstract:AI legal assistants based on Large Language Models (LLMs) can provide accessible legal consulting services, but the hallucination problem poses potential legal risks. This paper presents Chatlaw, an innovative legal assistant utilizing a Mixture-of-Experts (MoE) model and a multi-agent system to enhance the reliability and accuracy of AI-driven legal services. By integrating knowledge graphs with artificial screening, we construct a high-quality legal dataset to train the MoE model. This model utilizes different experts to address various legal issues, optimizing the accuracy of legal responses. Additionally, Standardized Operating Procedures (SOP), modeled after real law firm workflows, significantly reduce errors and hallucinations in legal services. Our MoE model outperforms GPT-4 in the Lawbench and Unified Qualification Exam for Legal Professionals by 7.73% in accuracy and 11 points, respectively, and also surpasses other models in multiple dimensions during real-case consultations, demonstrating our robust capability for legal consultation.

Computation and Language

What problem does this paper attempt to address?

The paper aims to address issues in the field of legal services, particularly the hallucination problem of large language models (LLMs) in legal consultations. Specifically, the paper proposes a multi-agent collaborative legal assistant system named Chatlaw, which utilizes a Mixture-of-Experts (MoE) model to enhance the reliability and accuracy of legal consultation services.

The main objectives of the paper include:
1. **Addressing the hallucination problem**: Traditional large language models tend to produce inaccurate or misleading information when dealing with legal issues, which is especially dangerous in the legal field. Chatlaw mitigates this problem through multi-layer filtering and a knowledge graph-enhanced dataset.
2. **Improving the quality of legal consultation**: Through a multi-agent collaborative framework, each agent is responsible for different tasks such as information gathering, in-depth research, legal advice, etc., ensuring the efficiency and accuracy of the consultation process.
3. **Optimizing model performance**: The paper demonstrates that Chatlaw outperforms existing large language models (such as GPT-4) in multiple benchmark tests (e.g., LawBench and the National Unified Legal Professional Qualification Examination), particularly in memory, understanding, and application of legal knowledge.

In summary, the core purpose of this paper is to develop a system that can effectively solve the hallucination problem in legal consultations and perform excellently in various legal tasks.