Abstract:Large vision-language models (VLMs) have recently achieved remarkable progress, exhibiting impressive multimodal perception and reasoning abilities. However, effectively evaluating these large VLMs remains a major challenge, hindering future development in this domain. Traditional benchmarks like VQAv2 or COCO Caption provide quantitative performance measurements but lack fine-grained ability assessment and robust evaluation metrics. Meanwhile, subjective benchmarks, such as OwlEval, offer comprehensive evaluations of a model's abilities by incorporating human labor, which is not scalable and may display significant bias. In response to these challenges, we propose MMBench, a bilingual benchmark for assessing the multi-modal capabilities of VLMs. MMBench methodically develops a comprehensive evaluation pipeline, primarily comprised of the following key features: 1. MMBench is meticulously curated with well-designed quality control schemes, surpassing existing similar benchmarks in terms of the number and variety of evaluation questions and abilities; 2. MMBench introduces a rigorous CircularEval strategy and incorporates large language models to convert free-form predictions into pre-defined choices, which helps to yield accurate evaluation results for models with limited instruction-following capabilities. 3. MMBench incorporates multiple-choice questions in both English and Chinese versions, enabling an apples-to-apples comparison of VLMs' performance under a bilingual context. To summarize, MMBench is a systematically designed objective benchmark for a robust and holistic evaluation of vision-language models. We hope MMBench will assist the research community in better evaluating their models and facilitate future progress in this area. The evalutation code of MMBench has been integrated into VLMEvalKit: <a class="link-external link-https" href="https://github.com/open-compass/VLMEvalKit" rel="external noopener nofollow">this https URL</a>.

Positive and Negative Critiquing for VAE-based Recommenders

Fast Multi-Step Critiquing for VAE-based Recommender Systems

VAE*: A Novel Variational Autoencoder via Revisiting Positive and Negative Samples for Top-N Recommendation

Critic-V: VLM Critics Help Catch VLM Errors in Multimodal Reasoning

Interacting with Explanations through Critiquing

Improving Reward Models with Synthetic Critiques

Self-Generated Critiques Boost Reward Modeling for Language Models

VISCO: Benchmarking Fine-Grained Critique and Correction Towards Self-Improvement in Visual Reasoning

Towards Flexible Evaluation for Generative Visual Question Answering

LLaVA-Critic: Learning to Evaluate Multimodal Models

ContrastVAE: Contrastive Variational AutoEncoder for Sequential Recommendation

VAEGAN: A Collaborative Filtering Framework based on Adversarial Variational Autoencoders

Self-critiquing models for assisting human evaluators

V-RECS, a Low-Cost LLM4VIS Recommender with Explanations, Captioning and Suggestions

MMBench: Is Your Multi-modal Model an All-around Player?

Self-supervised variational autoencoder towards recommendation by nested contrastive learning

Multimodal Review Generation for Recommender Systems

Task Success is not Enough: Investigating the Use of Video-Language Models as Behavior Critics for Catching Undesirable Agent Behaviors

VLRewardBench: A Challenging Benchmark for Vision-Language Generative Reward Models

CritiqueLLM: Towards an Informative Critique Generation Model for Evaluation of Large Language Model Generation

Enhancing Multimodal Compositional Reasoning of Visual Language Models with Generative Negative Mining