What problem does this paper attempt to address?

### Problems the Paper Aims to Solve The paper aims to address two main challenges faced by large language models (LLMs) when answering complex questions: 1. **Unfaithful Generation**: The responses from LLMs may not align with real-time or domain-specific facts. For example, in Figure 1(b), the LLM fails to locate relevant facts. 2. **Weak Reasoning Ability**: LLMs perform poorly in aggregating heterogeneous information sources, resolving conflicts, and providing useful and customized responses. For instance, in Figure 1(c), although relevant search results are successfully located, the analysis still halts. To tackle these challenges, the paper proposes a new framework—**Chain-of-Action (CoA)**, which decomposes complex questions into reasoning chains through systematic prompts and predefined actions. Specifically, the main contributions of the CoA framework include: - **Novel Reasoning-Retrieval Mechanism**: Decomposing complex questions into reasoning chains through systematic prompts and predefined actions. - **Three Pluggable Domain-Adaptive Actions**: Retrieving real-time information from heterogeneous sources, encoding domain knowledge, and analyzing tabular and numerical data. - **Multi-Reference Faithfulness Scoring (MRFS)**: Verifying and resolving conflicts in the answers. ### Experiments and Applications The paper demonstrates the capabilities of the CoA framework through public benchmarks and a Web3 case study. Experimental results show that the CoA framework outperforms existing methods on multiple QA datasets and achieves significant user engagement and positive feedback in practical applications. ### Main Contributions 1. **Proposing the CoA Framework**: Integrating a new reasoning-retrieval mechanism that decomposes complex questions into configurable action reasoning chains. 2. **Designing Three Pluggable Domain-Adaptive Actions**: For real-time information retrieval, domain knowledge encoding, and tabular data analysis. 3. **Introducing Multi-Reference Faithfulness Scoring (MRFS)**: Identifying and resolving conflicts between retrieved information and LLM-generated answers, enhancing answer reliability. 4. **Experimental Results**: The CoA framework outperforms existing methods in multiple public benchmarks. 5. **Practical Application**: Deploying the CoA framework in a Web3 QA application, significantly improving user engagement and positive feedback, validating its effectiveness and practicality in real-world scenarios. ### Methodology The paper details the methodology of the CoA framework, including how to generate action chains, execute actions, and ultimately generate answers. The specific steps are as follows: 1. **Action Chain Generation**: Generating action chains through context learning, with each action node containing sub-questions, missing flags, and initial answers. 2. **Action Execution and Monitoring**: Handling the multimodal retrieval needs of nodes through three steps: retrieving relevant information, verifying if the LLM-generated answers need correction, and filling in missing content if necessary. 3. **Final Answer Generation**: Generating the final answer by the LLM based on the refined and processed action chain. ### Experimental Analysis The paper presents a detailed experimental analysis, showcasing the performance advantages of the CoA framework on different QA datasets. Specifically, it shows: - **No Information Retrieval Tasks**: The CoA framework improves by 3.42% over the best existing baseline method (SearchChain without IR) in no information retrieval tasks. - **Information Retrieval Tasks**: The CoA framework improves by 6.14% over the best existing baseline method (SearchChain) in information retrieval tasks. - **Reasoning Steps**: The CoA framework averages more reasoning steps when decomposing complex questions, indicating stronger reasoning capabilities in handling complex problems. - **LLM Usage Frequency**: The CoA framework reduces the frequency of LLM usage, improving efficiency. - **Resistance to External Knowledge Interference**: The CoA framework demonstrates higher accuracy in handling external knowledge, indicating strong data parsing and filtering capabilities. ### Conclusion The paper empirically demonstrates the superiority of the CoA framework in understanding and answering complex queries, showcasing advanced reasoning capabilities and resistance to external erroneous information. These findings establish the CoA framework as a new benchmark in the fields of question answering and fact-checking, highlighting its comprehensive superiority.

Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models

Conv-CoA: Improving Open-domain Question Answering in Large Language Models via Conversational Chain-of-Action

CoQ:AN Empirical Framework for Multi-hop Question Answering Empowered by Large Language Models

Answering Questions by Meta-Reasoning over Multiple Chains of Thought

Multimodal Chain-of-Thought Reasoning in Language Models

Retrieval-Augmented Chain-of-Thought in Semi-structured Domains

Improving Complex Knowledge Base Question Answering via Question-to-Action and Question-to-Question Alignment

AQA: Adaptive Question Answering in a Society of LLMs via Contextual Multi-Armed Bandit

Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering

Tree-of-Reasoning Question Decomposition for Complex Question Answering with Large Language Models

Chain-of-Discussion: A Multi-Model Framework for Complex Evidence-Based Question Answering

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Leveraging Structured Information for Explainable Multi-hop Question Answering and Reasoning

AtomR: Atomic Operator-Empowered Large Language Models for Heterogeneous Knowledge Reasoning

Exploiting Reasoning Chains for Multi-hop Science Question Answering

Towards Faithful Chain-of-Thought: Large Language Models are Bridging Reasoners

LONGAGENT: Achieving Question Answering for 128K-Token-long Documents Through Multi-Agent Collaboration

Chain of Agents: Large Language Models Collaborating on Long-Context Tasks

Multi-hop Question Answering via Reasoning Chains

T-SciQ: Teaching Multimodal Chain-of-Thought Reasoning Via Large Language Model Signals for Science Question Answering.

Chain-of-Knowledge: Grounding Large Language Models via Dynamic Knowledge Adapting over Heterogeneous Sources