Abstract:Informal reasoning ability is the ability to reason based on common sense, experience, and intuition.Humans use informal reasoning every day to extract the most influential elements for their decision-making from a large amount of life-like information.With the rapid development of language models, the realization of general artificial intelligence has emerged with hope. Given the outstanding informal reasoning ability of humans, how much informal reasoning ability language models have has not been well studied by scholars.In order to explore the gap between humans and language models in informal reasoning ability, this paper constructs a Detective Reasoning Benchmark, which is an assembly of 1,200 questions gathered from accessible online resources, aims at evaluating the model's informal reasoning ability in real-life context.Considering the improvement of the model's informal reasoning ability restricted by the lack of benchmark, we further propose a Self-Question Prompt Framework that mimics human thinking to enhance the model's informal reasoning ability.The goals of self-question are to find key elements, deeply investigate the connections between these elements, encourage the relationship between each element and the problem, and finally, require the model to reasonably answer the problem.The experimental results show that human performance greatly outperforms the SoTA Language Models in Detective Reasoning Benchmark.Besides, Self-Question is proven to be the most effective prompt engineering in improving GPT-4's informal reasoning ability, but it still does not even surpass the lowest score made by human participants.Upon acceptance of the paper, the source code for the benchmark will be made publicly accessible.

The Study of Informal Reasoning and Its Prospect

Reasoning in Social Settings

Go Beyond The Obvious: Probing the gap of INFORMAL reasoning ability between Humanity and LLMs by Detective Reasoning Puzzle Benchmark

The development of reasoning skills during compulsory 16 to 18 mathematics education

Development of Children's Informal Mathematics in China

The Impact of Reasoning in Learning and Cognitive Processing

Reasoning in attitudes

ASSESSING STATISTICAL REASONING IN DESCRIPTIVE STATISTICS: A QUALITATIVE META-ANALYSIS

YOUNG CHILDREN’S PROBABILISTIC AND STATISTICAL REASONING IN THE CONTEXT OF INFORMAL STATISTICAL INFERENCE

College Students of Normal Colleges and Universities: Social Software of Informal Study Present Situation Investigation and Analysis of the Results

Investigating students' scientific reasoning through heuristic and analytical thought processes

Investigating students' reasoning in generalization of deconstructive figural patterns

Examining the unique contributions and developmental stability of individual forms of relational reasoning to mathematical problem solving

Navigate through Enigmatic Labyrinth A Survey of Chain of Thought Reasoning: Advances, Frontiers and Future

THE DEVELOPMENT OF FORMAL THINKING

Reasoning in artificial intelligence:advances and challenges

On Argument Evaluation from A Perspective of Informal Logic

A Survey of Reasoning with Foundation Models

A Study on the University Teachers' Practical Reasoning from the Chinese Socio-Cultural Perspective

The Cognitive Neural Mechanism of Human Inductive Reasoning: A Brain Informatics Perspective

Peano: Learning Formal Mathematical Reasoning