Abstract:In the intricate field of legal studies, the analysis of court decisions is a cornerstone for the effective functioning of the judicial system. The ability to predict court outcomes helps judges during the decision-making process and equips lawyers with invaluable insights, enhancing their strategic approaches to cases. Despite its significance, the domain of Arabic court analysis remains under-explored. This paper pioneers a comprehensive predictive analysis of Arabic court decisions on a dataset of 10,813 commercial court real cases, leveraging the advanced capabilities of the current state-of-the-art large language models. Through a systematic exploration, we evaluate three prevalent foundational models (LLaMA-7b, JAIS-13b, and GPT-3.5-turbo) and three training paradigms: zero-shot, one-shot, and tailored fine-tuning. In addition, we assess the benefit of summarizing and/or translating the original Arabic input texts. This leads to a spectrum of 14 model variants, for which we offer a granular performance assessment with a series of different metrics (human assessment, GPT evaluation, ROUGE, and BLEU scores). We show that all variants of LLaMA models yield limited performance, whereas GPT-3.5-based models outperform all other models by a wide margin, surpassing the average score of the dedicated Arabic-centric JAIS model by 50%. Furthermore, we show that all scores except human evaluation are inconsistent and unreliable for assessing the performance of large language models on court decision predictions. This study paves the way for future research, bridging the gap between computational linguistics and Arabic legal analytics.

AI Technologies in the Judiciary: Critical Appraisal of Large Language Models in Judicial Decision-making

Large Language Models in Law: A Survey

Legal Minds, Algorithmic Decisions: How LLMs Apply Constitutional Principles in Complex Scenarios

The Ethics of Automating Legal Actors

Caveat Lector: Large Language Models in Legal Practice

A Comprehensive Evaluation of Large Language Models on Legal Judgment Prediction

Artificial intelligence and judicial decision-making: Evaluating the role of AI in debiasing

Risk Assessment of Large Language Models Beyond Apocalyptic Visions

Attention is not all you need: the complicated case of ethically using large language models in healthcare and medicine

From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judge

A Survey on Large Language Models for Critical Societal Domains: Finance, Healthcare, and Law

AI Act and Large Language Models (LLMs): When critical issues and privacy impact require human and ethical oversight

Large Language Models in Politics and Democracy: A Comprehensive Survey

Navigating LLM Ethics: Advancements, Challenges, and Future Directions

LLMs instead of Human Judges? A Large Scale Empirical Study across 20 NLP Evaluation Tasks

Adapting to the Impact of AI in Scientific Writing: Balancing Benefits and Drawbacks while Developing Policies and Regulations

"I Always Felt that Something Was Wrong.": Understanding Compliance Risks and Mitigation Strategies when Professionals Use Large Language Models

Prediction of Arabic Legal Rulings Using Large Language Models

Optimizing Numerical Estimation and Operational Efficiency in the Legal Domain through Large Language Models

Large language models as tax attorneys: a case study in legal capabilities emergence