Abstract:Large language models (LLMs) have demonstrated limitations in handling combinatorial optimization problems involving long-range reasoning, partially due to causal hallucinations and huge search space. As for causal hallucinations, i.e., the inconsistency between reasoning and corresponding state transition, this paper introduces the Causal Relationship Enhancement (CRE) mechanism combining cause-effect interventions and the Individual Treatment Effect (ITE) to guarantee the solid causal rightness between each step of reasoning and state transition. As for the long causal range and huge search space limiting the performances of existing models featuring single-direction search, a Dual-End Searching (DES) approach is proposed to seek solutions by simultaneously starting from both the initial and goal states on the causal probability tree. By integrating CRE and DES (CreDes), our model has realized simultaneous multi-step reasoning, circumventing the inefficiencies from cascading multiple one-step reasoning like the Chain-of-Thought (CoT). Experiments demonstrate that CreDes significantly outperforms existing State-Of-The-Art (SOTA) solutions in long-range reasoning tasks in terms of both accuracy and time efficiency.

What problem does this paper attempt to address?

### What problems does this paper attempt to solve? This paper aims to solve two main challenges that large language models (LLMs) encounter when dealing with combinatorial optimization problems involving long - range reasoning: 1. **Causal Hallucinations**: - LLMs lack a strict causal relationship between performing one - step reasoning (OSR) and state transitions, resulting in unreliable reasoning results. Specifically, causal hallucinations refer to the inconsistency between the causal relationships in the reasoning process and the actual state transitions, which makes LLMs perform poorly in complex reasoning tasks. 2. **Huge Search Space**: - Long - range reasoning problems have a large state space. Traditional one - way search methods (such as Monte Carlo search) are inefficient in dealing with such large - scale search spaces and are prone to getting trapped in local optimal solutions, especially during multi - step reasoning processes. To solve these problems, the paper proposes two mechanisms: - **Causal Relationship Enhancement (CRE)**: - By introducing causal interventions and individual treatment effects (ITE), the CRE mechanism ensures that there is a solid causal relationship between each step of reasoning and its corresponding state transition, thereby reducing causal hallucinations. - **Dual - End Searching (DES)**: - The DES method decomposes long - range reasoning problems into multiple short - range reasoning problems by constructing causal probability trees starting from both the initial state and the target state simultaneously and finding the intersection paths between the two trees, thus improving search efficiency and accuracy. Combining these two mechanisms, the paper proposes a framework named CreDes, which can achieve higher accuracy and time efficiency in multi - step reasoning processes. ### Summary The core problem of this paper is to improve the performance of LLMs in long - range reasoning tasks, especially by addressing the challenges brought by causal hallucinations and huge search spaces. By introducing the CRE and DES mechanisms, the CreDes framework significantly improves the accuracy and efficiency of LLMs in long - range reasoning tasks.

CreDes: Causal Reasoning Enhancement and Dual-End Searching for Solving Long-Range Reasoning Problems using LLMs

Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.

CSCE: Boosting LLM Reasoning by Simultaneous Enhancing of Casual Significance and Consistency

Improving Causal Reasoning in Large Language Models: A Survey

Can We Further Elicit Reasoning in LLMs? Critic-Guided Planning with Retrieval-Augmentation for Solving Challenging Tasks

Q*: Improving Multi-step Reasoning for LLMs with Deliberative Planning

Causal Reasoning in Large Language Models: A Knowledge Graph Approach

Automatic Curriculum Expert Iteration for Reliable LLM Reasoning

Insight-V: Exploring Long-Chain Visual Reasoning with Multimodal Large Language Models

The Role of Deductive and Inductive Reasoning in Large Language Models

ReCo: Reliable Causal Chain Reasoning via Structural Causal Recurrent Neural Networks

Enhancing Event Causality Identification with Rationale and Structure-Aware Causal Question Answering

Deceptive Semantic Shortcuts on Reasoning Chains: How Far Can Models Go without Hallucination?

Experts Collaboration Learning for Continual Multi-Modal Reasoning

Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration

Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoning

ACRE: Abstract Causal REasoning Beyond Covariation

Causal-relationship representation enhanced joint extraction model for elements and relationships

Large Language Model for Causal Decision Making

An Enhanced Prompt-Based LLM Reasoning Scheme via Knowledge Graph-Integrated Collaboration

Learning a Structural Causal Model for Intuition Reasoning in Conversation