Abstract:Informal reasoning ability is the ability to reason based on common sense, experience, and intuition.Humans use informal reasoning every day to extract the most influential elements for their decision-making from a large amount of life-like information.With the rapid development of language models, the realization of general artificial intelligence has emerged with hope. Given the outstanding informal reasoning ability of humans, how much informal reasoning ability language models have has not been well studied by scholars.In order to explore the gap between humans and language models in informal reasoning ability, this paper constructs a Detective Reasoning Benchmark, which is an assembly of 1,200 questions gathered from accessible online resources, aims at evaluating the model's informal reasoning ability in real-life context.Considering the improvement of the model's informal reasoning ability restricted by the lack of benchmark, we further propose a Self-Question Prompt Framework that mimics human thinking to enhance the model's informal reasoning ability.The goals of self-question are to find key elements, deeply investigate the connections between these elements, encourage the relationship between each element and the problem, and finally, require the model to reasonably answer the problem.The experimental results show that human performance greatly outperforms the SoTA Language Models in Detective Reasoning Benchmark.Besides, Self-Question is proven to be the most effective prompt engineering in improving GPT-4's informal reasoning ability, but it still does not even surpass the lowest score made by human participants.Upon acceptance of the paper, the source code for the benchmark will be made publicly accessible.

Reasoning with Language Model Prompting: A Survey

Advances in reasoning by prompting large language models: A survey

A Survey of Reasoning with Foundation Models

Self-Polish: Enhance Reasoning in Large Language Models Via Problem Refinement.

Reasoning with Large Language Models, a Survey

Prompt Space Optimizing Few-shot Reasoning Success with Large Language Models

Go Beyond The Obvious: Probing the gap of INFORMAL reasoning ability between Humanity and LLMs by Detective Reasoning Puzzle Benchmark

Towards Better Chain-of-Thought Prompting Strategies: A Survey

Teaching-Inspired Integrated Prompting Framework: A Novel Approach for Enhancing Reasoning in Large Language Models

Towards Goal-oriented Prompt Engineering for Large Language Models: A Survey

Least-to-Most Prompting Enables Complex Reasoning in Large Language Models

Code Prompting: a Neural Symbolic Method for Complex Reasoning in Large Language Models

Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Models

Active Prompting with Chain-of-Thought for Large Language Models

Towards Reasoning in Large Language Models: A Survey

Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Models

RankPrompt: Step-by-Step Comparisons Make Language Models Better Reasoners

Efficient Prompting Methods for Large Language Models: A Survey

Hypothesis Testing Prompting Improves Deductive Reasoning in Large Language Models

Instance-adaptive Zero-shot Chain-of-Thought Prompting

Logic-of-Thought: Injecting Logic into Contexts for Full Reasoning in Large Language Models