Concise and Organized Perception Facilitates Large Language Models for Deductive Reasoning.
Junjie Liu,Shaotian Yan,Chen Shen,Liang Xie,Wenxiao Wang,Jieping Ye
DOI: https://doi.org/10.48550/arxiv.2310.03309
2023-01-01
Abstract:Exploiting large language models (LLMs) to tackle reasoning has garneredgrowing attention. It still remains highly challenging to achieve satisfactoryresults in complex logical problems, characterized by plenty of premises withinthe prompt and requiring multi-hop reasoning. In particular, the reasoningcapabilities of LLMs are brittle to disorder and distractibility. In this work,we first examine the mechanism from the perspective of information flow andreveal that LLMs exhibit failure patterns akin to human-like cognitive biaseswhen dealing with disordered and irrelevant content in reasoning tasks.However, in contrast to LLMs, disordered and irrelevant content does notsignificantly decrease human performance, as humans have a propensity todistill the most relevant information and systematically organize theirthoughts, aiding them in responding to questions. Stem from that, we furtherpropose a novel reasoning approach named Concise and Organized Perception(COP). COP carefully analyzes the given statements to identify the mostpertinent information while eliminating redundancy efficiently. It then promptsthe LLMs in a more organized form that adapts to the model's inference process.By perceiving concise and organized context, the reasoning abilities of LLMscan be better elicited. Extensive experimental results on several popularlogical benchmarks (ProofWriter, PrOntoQA, PrOntoQA-OOD, and FOLIO) and mathbenchmark (DI-GSM) show that COP significantly outperforms previousstate-of-the-art methods.