Abstract:Large vision-language models (VLMs) have recently achieved remarkable progress, exhibiting impressive multimodal perception and reasoning abilities. However, effectively evaluating these large VLMs remains a major challenge, hindering future development in this domain. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but lack fine-grained ability assessment and robust evaluation metrics. Meanwhile, subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, which is not scalable and may display significant bias. In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of the following key features: 1. MMBench is meticulously curated with well-designed quality control schemes, surpassing existing similar benchmarks in terms of the number and variety of evaluation questions and abilities; 2. MMBench introduces a rigorous CircularEval strategy and incorporates large language models to convert free-form predictions into pre-defined choices, which helps to yield accurate evaluation results for models with limited instruction-following capabilities. 3. MMBench incorporates multiple-choice questions in both English and Chinese versions, enabling an apples-to-apples comparison of VLMs' performance under a bilingual context. To summarize, MMBench is a systematically designed objective benchmark for a robust and holistic evaluation of vision-language models. We hope MMBench will assist the research community in better evaluating their models and facilitate future progress in this area. The evalutation code of MMBench has been integrated into VLMEvalKit: <a class="link-external link-https" href="https://github.com/open-compass/VLMEvalKit" rel="external noopener nofollow">this https URL</a>.

Value-Spectrum: Quantifying Preferences of Vision-Language Models via Value Decomposition in Social Media Contexts

Benchmarking Vision Language Models for Cultural Understanding

VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluation

ValueBench: Towards Comprehensively Evaluating Value Orientations and Understanding of Large Language Models

WildVision: Evaluating Vision-Language Models in the Wild with Human Preferences

VIVA: A Benchmark for Vision-Grounded Decision-Making with Human Values

VVA: Video Values Analysis.

WorldValuesBench: A Large-Scale Benchmark Dataset for Multi-Cultural Value Awareness of Language Models

Vision Language Models are In-Context Value Learners

Synthesize, Diagnose, and Optimize: Towards Fine-Grained Vision-Language Understanding

From Values to Opinions: Predicting Human Behaviors and Stances Using Value-Injected Large Language Models

MMBench: Is Your Multi-modal Model an All-around Player?

Right this way: Can VLMs Guide Us to See More to Answer Questions?

Guiding Vision-Language Model Selection for Visual Question-Answering Across Tasks, Domains, and Knowledge Types

ValueDCG: Measuring Comprehensive Human Value Understanding Ability of Language Models

Towards Flexible Evaluation for Generative Visual Question Answering

Visual Contexts Clarify Ambiguous Expressions: A Benchmark Dataset

Vision-Language Models under Cultural and Inclusive Considerations

ValueNet: A New Dataset for Human Value Driven Dialogue System

Beyond Human Norms: Unveiling Unique Values of Large Language Models through Interdisciplinary Approaches

VisionArena: 230K Real World User-VLM Conversations with Preference Labels