Abstract:Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at <a class="link-external link-https" href="https://infimm.github.io/InfiMM-Eval/" rel="external noopener nofollow">this https URL</a>

MTR: A Dataset Fusing Inductive, Deductive, and Defeasible Reasoning

Deriving Commonsense Inference Tasks from Interactive Fictions

Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification

LogiQA: A Challenge Dataset for Machine Reading Comprehension with Logical Reasoning

Enhancing human-like multimodal reasoning: a new challenging dataset and comprehensive framework

InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models

R3: A Reading Comprehension Benchmark Requiring Reasoning Processes

Abstract Reasoning with Distracting Features

BDIQA: A New Dataset for Video Question Answering to Explore Cognitive Reasoning through Theory of Mind

Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework

To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning.

RAVEN: A Dataset for Relational and Analogical Visual rEasoNing

Language Models as Inductive Reasoners

MIRAGE: Evaluating and Explaining Inductive Reasoning Process in Language Models

ConvoSense: Overcoming Monotonous Commonsense Inferences for Conversational AI

Towards Human-level Machine Reading Comprehension: Reasoning and Inference with Multiple Strategies.

MDCR: A Dataset for Multi-Document Conditional Reasoning

Boosting Deductive Reasoning with Step Signals In RLHF

RiTeK: A Dataset for Large Language Models Complex Reasoning over Textual Knowledge Graphs

Multi-Step Deductive Reasoning Over Natural Language: An Empirical Study on Out-of-Distribution Generalisation