AlignMMBench: Evaluating Chinese Multimodal Alignment in Large Vision-Language Models

Yuhang Wu,Wenmeng Yu,Yean Cheng,Yan Wang,Xiaohan Zhang,Jiazheng Xu,Ming Ding,Yuxiao Dong

2024-06-14

Abstract:Evaluating the alignment capabilities of large Vision-Language Models (VLMs) is essential for determining their effectiveness as helpful assistants. However, existing benchmarks primarily focus on basic abilities using nonverbal methods, such as yes-no and multiple-choice questions. In this paper, we address this gap by introducing AlignMMBench, a comprehensive alignment benchmark specifically designed for emerging Chinese VLMs. This benchmark is meticulously curated from real-world scenarios and Chinese Internet sources, encompassing thirteen specific tasks across three categories, and includes both single-turn and multi-turn dialogue scenarios. Incorporating a prompt rewrite strategy, AlignMMBench encompasses 1,054 images and 4,978 question-answer pairs. To facilitate the evaluation pipeline, we propose CritiqueVLM, a rule-calibrated evaluator that exceeds GPT-4's evaluation ability. Finally, we report the performance of representative VLMs on AlignMMBench, offering insights into the capabilities and limitations of different VLM architectures. All evaluation codes and data are available on <a class="link-external link-https" href="https://alignmmbench.github.io" rel="external noopener nofollow">this https URL</a>.

Computation and Language,Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem this paper attempts to address is that existing evaluation benchmarks mainly focus on basic capabilities and use non-linguistic methods (such as true/false questions and multiple-choice questions) to assess the alignment capabilities of large Visual-Language Models (VLMs), especially lacking in evaluations within a Chinese context. To fill this gap, the authors propose AlignMMBench, a comprehensive benchmark specifically designed to evaluate the alignment capabilities of emerging Chinese VLMs. AlignMMBench is meticulously curated from real-world scenarios and Chinese internet resources, covering 13 specific tasks, including single-turn and multi-turn dialogue scenarios, aiming to assess VLMs' performance in a Chinese context in more detail. Additionally, to facilitate the evaluation process, the authors also propose CritiqueVLM, a rule-calibrated evaluator whose assessment capabilities surpass those of GPT-4. Through these efforts, the paper provides insights into the capabilities and limitations of different VLM architectures, offering valuable resources for the research and development of Chinese VLMs.