Abstract:Multi-modal Large Language Models (MLLMs) are increasingly prominent in the field of artificial intelligence. These models not only excel in traditional vision-language tasks but also demonstrate impressive performance in contemporary multi-modal benchmarks. Although many of these benchmarks attempt to holistically evaluate MLLMs, they typically concentrate on basic reasoning tasks, often yielding only simple yes/no or multi-choice responses. These methods naturally lead to confusion and difficulties in conclusively determining the reasoning capabilities of MLLMs. To mitigate this issue, we manually curate a benchmark dataset specifically designed for MLLMs, with a focus on complex reasoning tasks. Our benchmark comprises three key reasoning categories: deductive, abductive, and analogical reasoning. The queries in our dataset are intentionally constructed to engage the reasoning capabilities of MLLMs in the process of generating answers. For a fair comparison across various MLLMs, we incorporate intermediate reasoning steps into our evaluation criteria. In instances where an MLLM is unable to produce a definitive answer, its reasoning ability is evaluated by requesting intermediate reasoning steps. If these steps align with our manual annotations, appropriate scores are assigned. This evaluation scheme resembles methods commonly used in human assessments, such as exams or assignments, and represents what we consider a more effective assessment technique compared with existing benchmarks. We evaluate a selection of representative MLLMs using this rigorously developed open-ended multi-step elaborate reasoning benchmark, designed to challenge and accurately measure their reasoning capabilities. The code and data will be released at <a class="link-external link-https" href="https://infimm.github.io/InfiMM-Eval/" rel="external noopener nofollow">this https URL</a>

Benchmarking Foundation Models on Exceptional Cases: Dataset Creation and Validation

Benchmarking Foundation Models with Language-Model-as-an-Examiner

GraphFM: A Comprehensive Benchmark for Graph Foundation Model

AGIEval: A Human-Centric Benchmark for Evaluating Foundation Models

Benchmarking Pathology Foundation Models: Adaptation Strategies and Scenarios

Evaluating the Efficacy of Foundational Models: Advancing Benchmarking Practices to Enhance Fine-Tuning Decision-Making

IsoBench: Benchmarking Multimodal Foundation Models on Isomorphic Representations

Benchmarking Language Model Creativity: A Case Study on Code Generation

BabelBench: An Omni Benchmark for Code-Driven Analysis of Multimodal and Multistructured Data

Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gap

Benchmarking Children's ASR with Supervised and Self-supervised Speech Foundation Models

FETA: Towards Specializing Foundation Models for Expert Task Applications

OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problems

New Benchmarks for Asian Facial Recognition Tasks: Face Classification with Large Foundation Models

The BiGGen Bench: A Principled Benchmark for Fine-grained Evaluation of Language Models with Language Models

Foundation Model Sherpas: Guiding Foundation Models through Knowledge and Reasoning

AutoBencher: Creating Salient, Novel, Difficult Datasets for Language Models

Text-space Graph Foundation Models: Comprehensive Benchmarks and New Insights

InfiMM-Eval: Complex Open-Ended Reasoning Evaluation For Multi-Modal Large Language Models

CHAMP: A Competition-level Dataset for Fine-Grained Analyses of LLMs' Mathematical Reasoning Capabilities

Benchmarking Generative Models on Computational Thinking Tests in Elementary Visual Programming