Towards Generalizable and Faithful Logic Reasoning over Natural Language via Resolution Refutation

Zhouhao Sun,Xiao Ding,Li Du,Bibo Cai,Jinglong Gao,Ting Liu,Qin Bing

2024-04-03

Abstract:Large language models (LLMs) have achieved significant performance in various natural language reasoning tasks. However, they still struggle with performing first-order logic reasoning over formal logical theories expressed in natural language. This is because the previous LLMs-based reasoning systems have the theoretical incompleteness issue. As a result, it can only address a limited set of simple reasoning problems, which significantly decreases their generalization ability. To address this issue, we propose a novel framework, named Generalizable and Faithful Reasoner (GFaiR), which introduces the paradigm of resolution refutation. Resolution refutation has the capability to solve all first-order logic reasoning problems by extending reasoning rules and employing the principle of proof by contradiction, so our system's completeness can be improved by introducing resolution refutation. Experimental results demonstrate that our system outperforms previous works by achieving state-of-the-art performances in complex scenarios while maintaining performances in simple scenarios. Besides, we observe that GFaiR is faithful to its reasoning process.

Artificial Intelligence,Computation and Language

What problem does this paper attempt to address?

The problem that this paper attempts to solve is that in natural language processing, existing large - language models (LLMs) have theoretical incompleteness when performing first - order logical reasoning, which causes them to be able to solve only a limited number of simple reasoning problems and cannot be generalized to more complex scenarios. Specifically, the paper points out that the current reasoning systems based on LLMs have the hallucination problem, that is, these models may generate incorrect intermediate reasoning steps and thus reach the final conclusion, which not only affects the accuracy of reasoning but also reduces the completeness of the system.

To overcome these problems, the paper proposes a new framework - Generalizable and Faithful Reasoner (GFaiR), which introduces resolution refutation. Resolution refutation can solve all first - order logical reasoning problems by expanding the reasoning rules and adopting the principle of proving contradictions, thereby improving the completeness and credibility of the system. The experimental results show that GFaiR outperforms previous work in complex scenarios, while also maintaining good performance in simple scenarios, and its reasoning process is trustworthy.