Abstract:Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at <a class="link-external link-https" href="https://infimm.github.io/InfiMM-Eval/" rel="external noopener nofollow">this https URL</a>

Testing The Reasoning Power For Nli Models With Annotated Multi-Perspective Entailment Dataset

Multi-Perspective Inferrer: Reasoning Sentences Relationship from Holistic Perspective

Discourse Marker Augmented Network with Reinforcement Learning for Natural Language Inference.

AMR4NLI: Interpretable and robust NLI measures from semantic graphs

Enhancing Systematic Decompositional Natural Language Inference Using Informal Logic

Are Machines Better at Complex Reasoning? Unveiling Human-Machine Inference Gaps in Entailment Verification

Identification of Entailment and Contradiction Relations between Natural Language Sentences: A Neurosymbolic Approach

Explainable Natural Language Inference Via Identifying Important Rationales

Inversive-Reasoning Augmentation for Natural Language Inference.

LoNLI: An Extensible Framework for Testing Diverse Logical Reasoning Capabilities for NLI

To What Extent Do Natural Language Understanding Datasets Correlate to Logical Reasoning? A Method for Diagnosing Logical Reasoning.

InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models

Neural Natural Logic Inference for Interpretable Question Answering.

Entangled Relations: Leveraging NLI and Meta-analysis to Enhance Biomedical Relation Extraction

Label-aware Debiased Causal Reasoning for Natural Language Inference

Do Natural Language Explanations Represent Valid Logical Arguments? Verifying Entailment in Explainable NLI Gold Standards

LLMs for Relational Reasoning: How Far are We?

Meta-Reasoning: Semantics-Symbol Deconstruction for Large Language Models

Exploring Factual Entailment with NLI: A News Media Study

Diagnosing the First-Order Logical Reasoning Ability Through LogicNLI

No Strong Feelings One Way or Another: Re-operationalizing Neutrality in Natural Language Inference