Abstract:As Large Language Models (LLMs) continue to exhibit remarkable performance in natural language understanding tasks, there is a crucial need to measure their ability for human-like multi-step logical reasoning. Existing logical reasoning evaluation benchmarks often focus primarily on simplistic single-step or multi-step reasoning with a limited set of inference rules. Furthermore, the lack of datasets for evaluating non-monotonic reasoning represents a crucial gap since it aligns more closely with human-like reasoning. To address these limitations, we propose Multi-LogiEval, a comprehensive evaluation dataset encompassing multi-step logical reasoning with various inference rules and depths. Multi-LogiEval covers three logic types--propositional, first-order, and non-monotonic--consisting of more than 30 inference rules and more than 60 of their combinations with various depths. Leveraging this dataset, we conduct evaluations on a range of LLMs including GPT-4, ChatGPT, Gemini-Pro, Yi, Orca, and Mistral, employing a zero-shot chain-of-thought. Experimental results show that there is a significant drop in the performance of LLMs as the reasoning steps/depth increases (average accuracy of ~68% at depth-1 to ~43% at depth-5). We further conduct a thorough investigation of reasoning chains generated by LLMs which reveals several important findings. We believe that Multi-LogiEval facilitates future research for evaluating and enhancing the logical reasoning ability of LLMs. Data is available at <a class="link-external link-https" href="https://github.com/Mihir3009/Multi-LogiEval" rel="external noopener nofollow">this https URL</a>.

MindMap: Constructing Evidence Chains for Multi-Step Reasoning in Large Language Models

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

Exploring the Role of Reasoning Structures for Constructing Proofs in Multi-Step Natural Language Reasoning with Large Language Models

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

Towards a Mechanistic Interpretation of Multi-Step Reasoning Capabilities of Language Models

Chain of Logic: Rule-Based Reasoning with Large Language Models

Bi-Chainer: Automated Large Language Models Reasoning with Bidirectional Chaining

Multimodal Chain-of-Thought Reasoning in Language Models

Concise and Organized Perception Facilitates Reasoning in Large Language Models

LLM Reasoners: New Evaluation, Library, and Analysis of Step-by-Step Reasoning with Large Language Models

The Impact of Reasoning Step Length on Large Language Models

Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models

Boosting Language Models Reasoning with Chain-of-Knowledge Prompting

Enhancing Zero-Shot Chain-of-Thought Reasoning in Large Language Models through Logic

Structured Event Reasoning with Large Language Models

Structure Guided Prompt: Instructing Large Language Model in Multi-Step Reasoning by Exploring Graph Structure of the Text

CLR-Fact: Evaluating the Complex Logical Reasoning Capability of Large Language Models over Factual Knowledge

Multi-LogiEval: Towards Evaluating Multi-Step Logical Reasoning Ability of Large Language Models

Thought-Like-Pro: Enhancing Reasoning of Large Language Models through Self-Driven Prolog-based Chain-of-Thought