Abstract:The popularity of multimodal large language models (MLLMs) has triggered a recent surge in research efforts dedicated to evaluating these models. Nevertheless, existing evaluation studies of MLLMs primarily focus on the comprehension and reasoning of unimodal (vision) content, neglecting performance evaluations in the domain of multimodal (vision-language) content understanding. Beyond multimodal reasoning, tasks related to multimodal content comprehension necessitate a profound understanding of multimodal contexts, achieved through the multimodal interaction to obtain a final answer. In this paper, we introduce a comprehensive assessment framework called MM-BigBench, which incorporates a diverse range of metrics to offer an extensive evaluation of the performance of various models and instructions across a wide spectrum of diverse multimodal content comprehension tasks. Consequently, our work complements research on the performance of MLLMs in multimodal comprehension tasks, achieving a more comprehensive and holistic evaluation of MLLMs. To begin, we employ the Best Performance metric to ascertain each model's performance upper bound on different datasets. Subsequently, the Mean Relative Gain metric offers an assessment of the overall performance of various models and instructions, while the Stability metric measures their sensitivity. Furthermore, previous research centers on evaluating models independently or solely assessing instructions, neglecting the adaptability between models and instructions. We propose the Adaptability metric to quantify the adaptability between models and instructions. Our paper evaluates a total of 20 language models (14 MLLMs) on 14 multimodal datasets spanning 6 tasks, with 10 instructions for each task, and derives novel insights. Our code will be released at <a class="link-external link-https" href="https://github.com/declare-lab/MM-BigBench" rel="external noopener nofollow">this https URL</a>.

Probing the limitations of multimodal language models for chemistry and materials research

ChemVLM: Exploring the Power of Multimodal Large Language Models in Chemistry Area

Are large language models superhuman chemists?

Scientific Language Modeling: A Quantitative Review of Large Language Models in Molecular Science

VisScience: An Extensive Benchmark for Evaluating K12 Educational Multi-modal Scientific Reasoning

MM-BigBench: Evaluating Multimodal Models on Multimodal Content Comprehension Tasks

A Survey on Multimodal Benchmarks: In the Era of Large AI Models

What can Large Language Models do in chemistry? A comprehensive benchmark on eight tasks

ChemDFM-X: Towards Large Multimodal Model for Chemistry

MIBench: Evaluating Multimodal Large Language Models over Multiple Images

A Molecular Multimodal Foundation Model Associating Molecule Graphs with Natural Language

AI Agents in Chemical Research: An Intelligent Research Assistant System

Multi-Modal Instruction-Tuning Small-Scale Language-and-Vision Assistant for Semiconductor Electron Micrograph Analysis

LMM Chemical Research with Document Retrieval

MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI

MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding

A Survey on Benchmarks of Multimodal Large Language Models

ScienceAgentBench: Toward Rigorous Assessment of Language Agents for Data-Driven Scientific Discovery

PCA-Bench: Evaluating Multimodal Large Language Models in Perception-Cognition-Action Chain

Bridging Text and Molecule: A Survey on Multimodal Frameworks for Molecule

ChemEval: A Comprehensive Multi-Level Chemical Evaluation for Large Language Models