LIME: Less Is More for MLLM Evaluation

King Zhu,Qianbo Zang,Shian Jia,Siwei Wu,Feiteng Fang,Yizhi Li,Shawn Gavin,Tuney Zheng,Jiawei Guo,Bo Li,Haoning Wu,Xingwei Qu,Jian Yang,Zachary Liu,Xiang Yue,J.H. Liu,Chenghua Lin,Min Yang,Shiwen Ni,Wenhao Huang,Ge Zhang

2024-10-14

Abstract:Multimodal Large Language Models (MLLMs) are evaluated on various benchmarks, such as image captioning, visual question answering, and reasoning. However, many of these benchmarks include overly simple or uninformative samples, complicating the effective distinction of different MLLMs' performance. Furthermore, evaluating models across numerous benchmarks incurs a significant computational burden. To address these issues, we propose LIME (Less Is More for MLLM Evaluation), a refined and efficient benchmark curated through a semi-automated pipeline. This pipeline filters out uninformative samples and eliminates answer leakage by focusing on tasks that necessitate image-based understanding. Our experiments indicate that LIME reduces the number of samples by 76% and evaluation time by 77%, while also providing a more effective means of distinguishing the capabilities of different models. Notably, we find that traditional automatic metrics, such as CIDEr, are inadequate for assessing MLLMs' captioning performance; excluding the caption task score yields a more accurate reflection of overall model performance. All code and data are available at <a class="link-external link-https" href="https://github.com/kangreen0210/LIME" rel="external noopener nofollow">this https URL</a>.

Computer Vision and Pattern Recognition,Artificial Intelligence

What problem does this paper attempt to address?

### Problems the paper attempts to solve

This paper aims to solve several key problems in the evaluation of Multimodal Large Language Models (MLLMs):

1. **Samples in existing benchmark tests are too simple or uninformative**:
- Current multimodal benchmark tests contain many relatively simple samples (for example, "How many chairs are there in the picture?"), and these samples cannot effectively distinguish the capabilities of different MLLMs.
- Some annotation problems lead to wrong samples, making these benchmark tests unable to fully reflect the performance differences between different MLLMs.

2. **Computational burden brought by evaluating multiple benchmark tests**:
- The current unified multimodal evaluation framework requires a large amount of computational resources because it needs to integrate evaluation data from various benchmark tests.
- Effective selection of evaluation data is ignored by current researchers, resulting in an inefficient evaluation process.

3. **Deficiencies of traditional automatic evaluation metrics**:
- Traditional automatic evaluation metrics (such as CIDEr) perform poorly in evaluating the image captioning tasks of MLLMs and cannot reasonably evaluate the performance of the model.
- Excluding the scores of image captioning tasks can more accurately reflect the overall performance of the model.

To solve these problems, the paper proposes LIME (Less Is More for MLLM Evaluation), which is a refined and efficient benchmark test set. It filters out uninformative samples through a semi - automatic pipeline and eliminates the problem of answer leakage. LIME can more effectively distinguish the capabilities of different MLLMs while significantly reducing the number of samples and evaluation time.