Abstract:Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at <a class="link-external link-https" href="https://infimm.github.io/InfiMM-Eval/" rel="external noopener nofollow">this https URL</a>

Interactive Model with Structural Loss for Language-based Abductive Reasoning

Abductive natural language inference by interactive model with structural loss

L2R²: Leveraging Ranking for Abductive Reasoning

Natural Language Inference Using Lstm Model With Sentence Fusion

Lost in Inference: Rediscovering the Role of Natural Language Inference for Large Language Models

Leveraging LLMs for Hypothetical Deduction in Logical Inference: A Neuro-Symbolic Approach

Asynchronous Deep Interaction Network for Natural Language Inference.

Enhancing Ethical Explanations of Large Language Models through Iterative Symbolic Refinement

Inversive-Reasoning Augmentation for Natural Language Inference.

A Systematic Analysis of Large Language Models as Soft Reasoners: The Case of Syllogistic Inferences

Can Small Language Models Help Large Language Models Reason Better?: LM-Guided Chain-of-Thought

Large language models are in-context semantic reasoners rather than symbolic reasoners

Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning

Inductive Linguistic Reasoning with Large Language Models

Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models

Enhanced Lstm For Natural Language Inference

Structured, flexible, and robust: benchmarking and improving large language models towards more human-like behavior in out-of-distribution reasoning tasks

Joint Abductive and Inductive Neural Logical Reasoning

Disentangling Memory and Reasoning Ability in Large Language Models

InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models