Abstract:Large language Models (LLMs) have achieved promising performance on arithmetic reasoning tasks by incorporating step-by-step chain-of-thought (CoT) prompting. However, LLMs face challenges in maintaining factual consistency during reasoning, exhibiting tendencies to condition overlooking, question misinterpretation, and condition hallucination over given problems. Existing methods use coarse-grained feedback (e.g., whether the answer is correct) to improve factual consistency. In this work, we propose RCoT (Reversing Chain-of-Thought), a novel method to improve LLMs' reasoning abilities by automatically detecting and rectifying factual inconsistency in LLMs, generated solutions. To detect factual inconsistency, RCoT first asks LLMs to reconstruct the problem based on generated solutions. Then fine-grained comparisons between the original problem and the reconstructed problem expose the factual inconsistency in the original solutions. To rectify the solution, RCoT formulates detected factual inconsistency into fine-grained feedback to guide LLMs in revising solutions. Experimental results demonstrate improvements of RCoT over standard CoT, Self-Consistency and Self-Refine across seven arithmetic datasets. Moreover, we find that manually written fine-grained feedback can dramatically improve LLMs' reasoning abilities (e.g., ChatGPT reaches 94.6% accuracy on GSM8K), encouraging the community to further explore the fine-grained feedback generation methods.

A Non-revision Reasoning Method for Inconsistent Clausal Beliefs

Non-revision Reasoning with Beliefs in Predicate Logic.

Non-Revision Reasoning With Inconsistent Ontology

Argumentation-Based Non-Monotonic Reasoning of Agents

Non-revision Reasoning Without Redundancy in Predicate Logic

Non-monotonic Reasoning and Argumentation System

Minimally Inconsistent Reasoning in Semantic Web.

A logic for reasoning with inconsistent knowledge -- A reformulation using nowadays terminology (2024)

An Algorithm for Extension of Clausal Beliefs.

Argumentation-Based Reasoning With Inconsistent Knowledge Bases

Research on robot commonsense reasoning based on improved active logic and metacognitive loop

An Approach to Robot Commonsense Reasoning

A logic for default reasoning

An Up to Down Reasoning for Inconsistent Beliefs

Two Kinds of Reasoning with Credibility

A Logical Approach to Doxastic Causal Reasoning.

RCOT: Detecting and Rectifying Factual Inconsistency in Reasoning by Reversing Chain-of-Thought

A Logic for Paraconsistent Belief Revision based on Epistemic Entrenchment

Commonsense Reasoning in a Deeper Way: By Discovering Relations between Predicates.

A Logic for Iterated Belief Revision

Extending Automated Deduction for Commonsense Reasoning