Abstract:In many legal processes being able to action on the concrete implication of a legal question can be valuable to automating human review or signalling certain conditions (e.g., alerts around automatic renewal). To support such tasks, we present a form of legal question answering that seeks to return one (or more) fixed answers for a question about a contract clause. After showing that unstructured generative question answering can have questionable outcomes for such a task, we discuss our exploration methodology for legal question answering prompts using OpenAI's \textit{GPT-3.5-Turbo} and provide a summary of insights. Using insights gleaned from our qualitative experiences, we compare our proposed template prompts against a common semantic matching approach and find that our prompt templates are far more accurate despite being less reliable in the exact response return. With some additional tweaks to prompts and the use of in-context learning, we are able to further improve the performance of our proposed strategy while maximizing the reliability of responses as best we can.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is to automatically generate structured answers in legal documents. Specifically, the authors explored how to use generative AI technology, especially OpenAI's GPT - 3.5 - Turbo model, to generate specific answers regarding contract clauses (such as in the form of multiple - choice questions), rather than free - form responses. Such structured answers can be more conveniently applied to business process automation without the need for complex understanding and manual review. However, existing open - generation methods have problems of poor consistency, low reliability, and verbosity when answering such specific questions. Therefore, the main objective of the paper is to improve the accuracy and reliability of generative AI in handling legal documents by exploring and optimizing prompt engineering methods. ### Main Research Questions 1. **Improving the Accuracy of Generative AI in Legal Documents**: How can effective prompt templates be designed so that generative AI can provide accurate answers to specific legal questions? 2. **Enhancing the Reliability of Generative AI**: How can the consistency and reliability of generative AI under different contract clauses be ensured, avoiding significant differences in generation results due to minor changes in prompts? 3. **Reducing Redundant Information of Generative AI**: How can generative AI be prevented from generating unnecessary explanations or additional information when answering questions, thereby simplifying the subsequent processing flow? ### Solutions The authors solve the above problems through the following methods: 1. **Design of Prompt Templates**: Through testing a large number of combinations of prompts and contract clauses, the authors discovered some key principles for designing prompt templates, such as: - **No Explanation**: Clearly instruct the model not to provide explanations or additional information to reduce redundancy. - **Option Selection**: Specify that the answer can only be selected from predefined options, which helps guide the model to select the correct answer. - **Completeness of Options**: Ensure that each option is a complete sentence or idea to prevent the model from generating additional text. - **Escape Mechanism**: Provide an "undetermined" option. When the model lacks sufficient information, it can select this option. 2. **Experimental Verification**: The authors used generative AI models such as OpenAI's GPT - 3.5 - Turbo and GPT - 4, and Google's PaLM2 to conduct extensive tests on the designed prompt templates. They compared the performance of these models in generating structured answers and contrasted them with traditional semantic matching methods. ### Experimental Results - **Accuracy**: By optimizing the prompt templates, the accuracy of generative AI in answering legal questions has been significantly improved. - **Reliability**: The optimization of prompt templates has reduced the problems of consistency and reliability in generation results. - **Structured Answers**: The generated answers are more concise, reducing unnecessary explanations and redundant information. ### Conclusion The paper shows that through carefully designed prompt templates, generative AI can generate structured answers in legal documents, thereby better supporting business process automation. Although there are still some challenges, this method provides a valuable reference for future research.

A Search for Prompts: Generating Structured Answers from Contracts

Answering Questions in Stages: Prompt Chaining for Contract QA

Leveraging Event Schema to Ask Clarifying Questions for Conversational Legal Case Retrieval

Towards Understanding Contracts Grammar: A Large Language Model-Based Extractive Question-Answering Approach

Requirements Engineering using Generative AI: Prompts and Prompting Patterns

Weaving Pathways for Justice with GPT: LLM-driven automated drafting of interactive legal applications

Question-Analysis Prompting Improves LLM Performance in Reasoning Tasks

Performance analysis of large language models in the domain of legal argument mining

Generating Clarification Questions for Disambiguating Contracts

Experimenting with Legal AI Solutions: The Case of Question-Answering for Access to Justice

Team UTSA-NLP at SemEval 2024 Task 5: Prompt Ensembling for Argument Reasoning in Civil Procedures with GPT4

Autonomous Prompt Engineering in Large Language Models

Does Prompt Formatting Have Any Impact on LLM Performance?

The art of prompts' formulation: limitations, potential, and practical examples in large language models

A Prompt Engineering Approach and a Knowledge Graph based Framework for Tackling Legal Implications of Large Language Model Answers

Measuring the Groundedness of Legal Question-Answering Systems

Improving ChatGPT Prompt for Code Generation

LangGPT: Rethinking Structured Reusable Prompt Design Framework for LLMs from the Programming Language

PromptAgent: Strategic Planning with Language Models Enables Expert-level Prompt Optimization

Minstrel: Structural Prompt Generation with Multi-Agents Coordination for Non-AI Experts

Promptor: A Conversational and Autonomous Prompt Generation Agent for Intelligent Text Entry Techniques