Abstract:Rule-based reasoning, a fundamental type of legal reasoning, enables us to draw conclusions by accurately applying a rule to a set of facts. We explore causal language models as rule-based reasoners, specifically with respect to compositional rules - rules consisting of multiple elements which form a complex logical expression. Reasoning about compositional rules is challenging because it requires multiple reasoning steps, and attending to the logical relationships between elements. We introduce a new prompting method, Chain of Logic, which elicits rule-based reasoning through decomposition (solving elements as independent threads of logic), and recomposition (recombining these sub-answers to resolve the underlying logical expression). This method was inspired by the IRAC (Issue, Rule, Application, Conclusion) framework, a sequential reasoning approach used by lawyers. We evaluate chain of logic across eight rule-based reasoning tasks involving three distinct compositional rules from the LegalBench benchmark and demonstrate it consistently outperforms other prompting methods, including chain of thought and self-ask, using open-source and commercial language models.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is how to use large - scale language models for rule - based reasoning, especially dealing with complex combinatorial rules. Rule - based reasoning is particularly important in the legal field because it involves making judgments on facts according to specific rules to draw conclusions. However, existing language models have difficulties in handling complex rules, especially in cases where multi - step reasoning and attention to logical relationships are required. Therefore, the author introduced a new prompting method - "Chain of Logic", aiming to improve the reasoning ability of the model by decomposing and recombining rule elements. Specifically, the following aspects are explored in the paper: 1. **Existing Challenges**: Existing language models perform poorly in handling complex rules in the legal field, especially in cases where multi - step reasoning and understanding of logical relationships are required. Even powerful models such as GPT - 4 perform poorly on basic legal tasks. 2. **New Method**: The author proposed the "Chain of Logic" method, which guides the model to perform rule - based reasoning through the following steps: - **Structured Input**: Clearly label the input of the task, including rules, facts, and questions. - **Rule Decomposition**: Decompose the rules into multiple elements and assign these elements to variables. - **Logical Expression**: Construct a complex logical expression representing the logical relationships between these elements. - **Question Answering**: Traverse each rule element, re - phrase it as a question, and provide reasons and answers. - **Element Recombination**: Replace the sub - answers in the previous step into the logical expression. - **Parse Expression**: Parse the logical expression filled with sub - answers to draw the final conclusion. 3. **Experimental Verification**: The author evaluated the "Chain of Logic" method in eight rule - based reasoning tasks, which involve three different combinatorial rules from the LegalBench benchmark. The experimental results show that the "Chain of Logic" method significantly outperforms other prompting methods in multiple tasks, including the "Chain of Thought" and "Self - Ask" methods. 4. **Performance Improvement**: Especially in tasks requiring arithmetic reasoning, the "Chain of Logic" method performs excellently and significantly improves the reasoning ability of the model. In conclusion, this paper aims to improve the reasoning ability of large - scale language models in handling complex combinatorial rules through the "Chain of Logic" method, thereby achieving more accurate and efficient reasoning in the legal field.

Chain of Logic: Rule-Based Reasoning with Large Language Models

Chain of Logic: Rule-Based Reasoning with Large Language Models

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models

LogicBench: Towards Systematic Evaluation of Logical Reasoning Ability of Large Language Models

Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought

Reasoning with Large Language Models, a Survey

Can LLMs Reason with Rules? Logic Scaffolding for Stress-Testing and Improving LLMs

Concise and Organized Perception Facilitates Reasoning in Large Language Models

A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences

Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic

CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge

Chain-of-Thought Prompting Elicits Reasoning in Large Language Models

Break the Chain: Large Language Models Can be Shortcut Reasoners

Logic Agent: Enhancing Validity with Logic Rule Invocation

LogicGame: Benchmarking Rule-Based Reasoning Abilities of Large Language Models

MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models

Chain-of-Knowledge: Integrating Knowledge Reasoning into Large Language Models by Learning from Knowledge Graphs

Bi-Chainer: Automated Large Language Models Reasoning with Bidirectional Chaining

Rulebreakers Challenge: Revealing a Blind Spot in Large Language Models' Reasoning with Formal Logic