Abstract:Informal reasoning ability is the ability to reason based on common sense, experience, and intuition.Humans use informal reasoning every day to extract the most influential elements for their decision-making from a large amount of life-like information.With the rapid development of language models, the realization of general artificial intelligence has emerged with hope. Given the outstanding informal reasoning ability of humans, how much informal reasoning ability language models have has not been well studied by scholars.In order to explore the gap between humans and language models in informal reasoning ability, this paper constructs a Detective Reasoning Benchmark, which is an assembly of 1,200 questions gathered from accessible online resources, aims at evaluating the model's informal reasoning ability in real-life context.Considering the improvement of the model's informal reasoning ability restricted by the lack of benchmark, we further propose a Self-Question Prompt Framework that mimics human thinking to enhance the model's informal reasoning ability.The goals of self-question are to find key elements, deeply investigate the connections between these elements, encourage the relationship between each element and the problem, and finally, require the model to reasonably answer the problem.The experimental results show that human performance greatly outperforms the SoTA Language Models in Detective Reasoning Benchmark.Besides, Self-Question is proven to be the most effective prompt engineering in improving GPT-4's informal reasoning ability, but it still does not even surpass the lowest score made by human participants.Upon acceptance of the paper, the source code for the benchmark will be made publicly accessible.

Probing Commonsense Reasoning Capability of Text-to-Image Generative Models Via Non-visual Description

Commonsense-T2I Challenge: Can Text-to-Image Generation Models Understand Commonsense?

CommonGen: A Constrained Text Generation Challenge for Generative Commonsense Reasoning

ROME: Evaluating Pre-trained Vision-Language Models on Reasoning beyond Visual Common Sense

From Recognition to Cognition: Visual Commonsense Reasoning

DALL-Eval: Probing the Reasoning Skills and Social Biases of Text-to-Image Generation Models

MORE: Multi-mOdal REtrieval Augmented Generative Commonsense Reasoning

Deriving Commonsense Inference Tasks from Interactive Fictions

CommonsenseVIS: Visualizing and Understanding Commonsense Reasoning Capabilities of Natural Language Models

UNcommonsense Reasoning: Abductive Reasoning about Uncommon Situations

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Commonsense Reasoning for Conversational AI: A Survey of the State of the Art

Zero-shot Commonsense Reasoning over Machine Imagination

Towards Reasoning Ability in Scene Text Visual Question Answering

Evaluating the Capabilities of Multi-modal Reasoning Models with Synthetic Task Data

Beyond Language: Learning Commonsense from Images for Reasoning

ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models

Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog

VisualCOMET: Reasoning about the Dynamic Context of a Still Image

Modeling Gestalt Visual Reasoning on the Raven's Progressive Matrices Intelligence Test Using Generative Image Inpainting Techniques

Go Beyond The Obvious: Probing the gap of INFORMAL reasoning ability between Humanity and LLMs by Detective Reasoning Puzzle Benchmark