Abstract:Answering complex questions is a challenging task that requires question decomposition and multistep reasoning for arriving at the solution. While existing supervised and unsupervised approaches are specialized to a certain task and involve training, recently proposed prompt-based approaches offer generalizable solutions to tackle a wide variety of complex question-answering (QA) tasks. However, existing prompt-based approaches that are effective for complex QA tasks involve expensive hand annotations from experts in the form of rationales and are not generalizable to newer complex QA scenarios and tasks. We propose, icat (In-Context Ability Transfer) which induces reasoning capabilities in LLMs without any LLM fine-tuning or manual annotation of in-context samples. We transfer the ability to decompose complex questions to simpler questions or generate step-by-step rationales to LLMs, by careful selection from available data sources of related tasks. We also propose an automated uncertainty-aware exemplar selection approach for selecting examples from transfer data sources. Finally, we conduct large-scale experiments on a variety of complex QA tasks involving numerical reasoning, compositional complex QA, and heterogeneous complex QA which require decomposed reasoning. We show that ICAT convincingly outperforms existing prompt-based solutions without involving any model training, showcasing the benefits of re-using existing abilities.

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: How can large - language models (LLMs) be enabled to decompose complex problems through in - context ability transfer (ICAT) without expensive fine - tuning or manual annotation, so as to effectively solve a variety of complex question - answering tasks? Specifically, although the existing prompt - based methods can handle complex question - answering tasks, they rely on reasons or annotations provided by experts, which are not only costly but also difficult to generalize to new complex question - answering scenarios. The paper proposes a new method - ICAT, which automatically constructs prompts by selecting examples of relevant tasks from existing datasets, thereby inducing the LLM to perform multi - step reasoning and problem decomposition without fine - tuning the LLM or manual annotation. ### Main contributions: 1. **Proposed a new method, ICAT**: This method re - utilizes the reasons in similar - ability datasets to achieve effective chain - of - thought reasoning. 2. **Proposed a new uncertainty - aware example - selection method**: This method can automatically select relevant examples without human intervention, thereby achieving competitive performance in dynamic - prompt construction. 3. **Conducted extensive experiments**: Demonstrated the generalization performance of ICAT in multiple complex question - answering tasks through different example - selection methods. ### Problems solved: - **Reduce the cost of manual annotation**: By automatically selecting examples, the need to manually annotate a large number of samples is avoided. - **Improve the generalization ability of the model**: ICAT can handle a variety of complex question - answering tasks without fine - tuning. - **Enhance the interpretability of the model**: Through chain - of - thought reasoning, the interpretability and accuracy of the model in solving complex problems are improved. ### Technical details: - **Problem decomposition**: ICAT induces the LLM to decompose complex problems into simpler sub - problems by selecting examples of relevant tasks and gradually reasons out the final answer. - **Dynamic example selection**: Proposed the Frechet Term Embedding Distance (FTD) method for selecting high - quality and diverse examples, ensuring that the selected examples are similar to and not redundant with the test samples. - **Experimental verification**: Conducted large - scale experiments on multiple complex question - answering datasets to verify the effectiveness and superiority of ICAT. In conclusion, the paper proposes an efficient and general - purpose method that can improve the performance of the LLM in complex question - answering tasks without incurring additional costs.

In-Context Ability Transfer for Question Decomposition in Complex QA

Tree-of-Reasoning Question Decomposition for Complex Question Answering with Large Language Models

DEXTER: A Benchmark for open-domain Complex Question Answering using LLMs

CoQ:AN Empirical Framework for Multi-hop Question Answering Empowered by Large Language Models

Enhancing the Completeness of Rationales for Multi-Step Question Answering

Program Transfer for Answering Complex Questions over Knowledge Bases

Question Decomposition Tree for Answering Complex Questions over Knowledge Bases

AQA: Adaptive Question Answering in a Society of LLMs via Contextual Multi-Armed Bandit

Question-Analysis Prompting Improves LLM Performance in Reasoning Tasks

GenSco: Can Question Decomposition based Passage Alignment improve Question Answering?

Unsupervised Question Decomposition for Question Answering

CuriousLLM: Elevating Multi-Document QA with Reasoning-Infused Knowledge Graph Prompting

A Step Closer to Comprehensive Answers: Constrained Multi-Stage Question Decomposition with Large Language Models

MFORT-QA: Multi-hop Few-shot Open Rich Table Question Answering

STOC-TOT: Stochastic Tree-of-Thought with Constrained Decoding for Complex Reasoning in Multi-Hop Question Answering

Thought Propagation: An Analogical Approach to Complex Reasoning with Large Language Models

Prompting Explicit and Implicit Knowledge for Multi-hop Question Answering Based on Human Reading Process

Seek and Solve Reasoning for Table Question Answering

Active Prompting with Chain-of-Thought for Large Language Models

Self-prompted Chain-of-Thought on Large Language Models for Open-domain Multi-hop Reasoning

Retrieval-Augmented Chain-of-Thought in Semi-structured Domains