AutoReason: Automatic Few-Shot Reasoning Decomposition

Arda Sevinc,Abdurrahman Gumus

2024-12-10

Abstract:Chain of Thought (CoT) was introduced in recent research as a method for improving step-by-step reasoning in Large Language Models. However, CoT has limited applications such as its need for hand-crafted few-shot exemplar prompts and no capability to adjust itself to different queries. In this work, we propose a system to automatically generate rationales using CoT. Our method improves multi-step implicit reasoning capabilities by decomposing the implicit query into several explicit questions. This provides interpretability for the model, improving reasoning in weaker LLMs. We test our approach with two Q\&A datasets: StrategyQA and HotpotQA. We show an increase in accuracy with both, especially on StrategyQA. To facilitate further research in this field, the complete source code for this study has been made publicly available on GitHub: <a class="link-external link-https" href="https://github.com/miralab-ai/autoreason" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Artificial Intelligence

What problem does this paper attempt to address?

The problem that this paper attempts to solve is: how to improve the performance of large - language models in tasks requiring multi - step reasoning, especially in the case of zero - shot prompting. Although the existing Chain of Thought (CoT) method can improve the reasoning ability of models, it depends on a small number of manually - designed few - shot exemplars, which limits its scalability and applicability in new fields and tasks. In addition, existing methods usually use a fixed set of examples to handle all queries, which may lead to the generated reasoning paths being not specific or relevant enough, thus affecting performance.

For this reason, the authors propose the AutoReason system. This system enhances the multi - step implicit reasoning ability of weaker language models by automatically generating reasoning paths (rationales) from zero - shot prompts and converting these paths into few - shot example reasoning traces. Specifically, the main research questions of AutoReason include:

1. Can the accuracy of zero - shot prompting be improved by generating reasoning paths?
2. Can a method be developed to use CoT to automatically generate reasoning paths to improve the multi - step implicit reasoning ability of weaker large - language models?
3. How can unique reasoning paths be generated for each query instead of relying on fixed CoT prompts to enhance the specificity and relevance of reasoning paths?
4. Can it be proven that the automatically generated reasoning paths are effective in improving the performance of weaker large - language models in complex multi - step reasoning tasks?

By solving these problems, AutoReason aims to promote technological progress in the field of large - language model reasoning, make CoT prompts more scalable and flexible, and lower the application threshold to make them more suitable for practical scenarios.