Go Beyond The Obvious: Probing the gap of INFORMAL reasoning ability
between Humanity and LLMs by Detective Reasoning Puzzle Benchmark
Zhouhong Gu,Zechuan Li,Lin Zhang,Zhuozhi Xiong,Suhua Jiang,Xiaomin Zhu,Shusen Wang,Zili Wang,Jianchen Wang,Haoning Ye,Wenhao Huang,Yikai Zhang,Hui Feng,Yanghua Xiao
DOI: https://doi.org/10.48550/arxiv.2307.05113
2023-01-01
Abstract:Informal reasoning ability is the ability to reason based on common sense, experience, and intuition.Humans use informal reasoning every day to extract the most influential elements for their decision-making from a large amount of life-like information.With the rapid development of language models, the realization of general artificial intelligence has emerged with hope. Given the outstanding informal reasoning ability of humans, how much informal reasoning ability language models have has not been well studied by scholars.In order to explore the gap between humans and language models in informal reasoning ability, this paper constructs a Detective Reasoning Benchmark, which is an assembly of 1,200 questions gathered from accessible online resources, aims at evaluating the model's informal reasoning ability in real-life context.Considering the improvement of the model's informal reasoning ability restricted by the lack of benchmark, we further propose a Self-Question Prompt Framework that mimics human thinking to enhance the model's informal reasoning ability.The goals of self-question are to find key elements, deeply investigate the connections between these elements, encourage the relationship between each element and the problem, and finally, require the model to reasonably answer the problem.The experimental results show that human performance greatly outperforms the SoTA Language Models in Detective Reasoning Benchmark.Besides, Self-Question is proven to be the most effective prompt engineering in improving GPT-4's informal reasoning ability, but it still does not even surpass the lowest score made by human participants.Upon acceptance of the paper, the source code for the benchmark will be made publicly accessible.