Abstract:Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at <a class="link-external link-https" href="https://infimm.github.io/InfiMM-Eval/" rel="external noopener nofollow">this https URL</a>

Enhancing human-like multimodal reasoning: a new challenging dataset and comprehensive framework

Enhancing Human-like Multi-Modal Reasoning: A New Challenging Dataset and Comprehensive Framework

Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answering

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

A Multi-Modal Context Reasoning Approach for Conditional Inference on Joint Textual and Visual Clues

Towards a Unified Multimodal Reasoning Framework

Multimodal Chain-of-Thought Reasoning in Language Models

DDCoT: Duty-Distinct Chain-of-Thought Prompting for Multimodal Reasoning in Language Models

Visual CoT: Unleashing Chain-of-Thought Reasoning in Multi-Modal Language Models

VoCoT: Unleashing Visually Grounded Multi-Step Reasoning in Large Multi-Modal Models

Boosting the Power of Small Multimodal Reasoning Models to Match Larger Models with Self-Consistency Training

MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale

M^3CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought

MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmark

CMMU: A Benchmark for Chinese Multi-modal Multi-type Question Understanding and Reasoning

M$^3$CoT: A Novel Benchmark for Multi-Domain Multi-step Multi-modal Chain-of-Thought

CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs

VideoCoT: A Video Chain-of-Thought Dataset with Active Annotation Tool

MiCEval: Unveiling Multimodal Chain of Thought's Quality via Image Description and Reasoning Steps

InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models

Multi-modal Latent Space Learning for Chain-of-Thought Reasoning in Language Models