LiveXiv -- A Multi-Modal Live Benchmark Based on Arxiv Papers Content

Nimrod Shabtay,Felipe Maia Polo,Sivan Doveh,Wei Lin,M. Jehanzeb Mirza,Leshem Chosen,Mikhail Yurochkin,Yuekai Sun,Assaf Arbelle,Leonid Karlinsky,Raja Giryes

2024-10-15

Abstract:The large-scale training of multi-modal models on data scraped from the web has shown outstanding utility in infusing these models with the required world knowledge to perform effectively on multiple downstream tasks. However, one downside of scraping data from the web can be the potential sacrifice of the benchmarks on which the abilities of these models are often evaluated. To safeguard against test data contamination and to truly test the abilities of these foundation models we propose LiveXiv: A scalable evolving live benchmark based on scientific ArXiv papers. LiveXiv accesses domain-specific manuscripts at any given timestamp and proposes to automatically generate visual question-answer pairs (VQA). This is done without any human-in-the-loop, using the multi-modal content in the manuscripts, like graphs, charts, and tables. Moreover, we introduce an efficient evaluation approach that estimates the performance of all models on the evolving benchmark using evaluations of only a subset of models. This significantly reduces the overall evaluation cost. We benchmark multiple open and proprietary Large Multi-modal Models (LMMs) on the first version of our benchmark, showing its challenging nature and exposing the models true abilities, avoiding contamination. Lastly, in our commitment to high quality, we have collected and evaluated a manually verified subset. By comparing its overall results to our automatic annotations, we have found that the performance variance is indeed minimal (<2.5%). Our dataset is available online on HuggingFace, and our code will be available here.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

This paper attempts to address the issue of test set contamination caused by data scraped from the web during large-scale training of multimodal models, which can affect our understanding of these models' capabilities. To tackle this challenge, the paper proposes LiveXiv, a dynamic multimodal benchmarking method based on the content of ArXiv papers. Specifically:

1. **Preventing Data Contamination**: By continuously updating the benchmark, it avoids the test set contamination issues that static datasets might cause, ensuring the authenticity of evaluation results.

2. **Automatic Generation of Multimodal Data**: It automatically extracts information such as charts from scientific papers on ArXiv and generates visual question answering (VQA) pairs without human intervention.

3. **Efficient Evaluation Mechanism**: An efficient evaluation framework is proposed, which allows inferring the model's performance on the latest version of the benchmark by re-evaluating only a portion of the data, significantly reducing computational costs.

4. **Ensuring Quality and Challenge**: Through rigorous filtering stages, it reduces errors and maintains the high quality and challenge of the dataset, ensuring it accurately reflects the capabilities of multimodal models.

In summary, this paper aims to provide an ever-evolving, contamination-free, and efficient multimodal model evaluation solution through the innovative dynamic benchmarking method of LiveXiv.