Benchmarking Sequential Visual Input Reasoning and Prediction in Multimodal Large Language Models

Mingwei Zhu,Leigang Sha,Yu Shu,Kangjia Zhao,Tiancheng Zhao,Jianwei Yin

DOI: https://doi.org/10.48550/arXiv.2310.13473

2023-10-20

Abstract:Multimodal large language models (MLLMs) have shown great potential in perception and interpretation tasks, but their capabilities in predictive reasoning remain under-explored. To address this gap, we introduce a novel benchmark that assesses the predictive reasoning capabilities of MLLMs across diverse scenarios. Our benchmark targets three important domains: abstract pattern reasoning, human activity prediction, and physical interaction prediction. We further develop three evaluation methods powered by large language model to robustly quantify a model's performance in predicting and reasoning the future based on multi-visual context. Empirical experiments confirm the soundness of the proposed benchmark and evaluation methods via rigorous testing and reveal pros and cons of current popular MLLMs in the task of predictive reasoning. Lastly, our proposed benchmark provides a standardized evaluation framework for MLLMs and can facilitate the development of more advanced models that can reason and predict over complex long sequence of multimodal input.

Computer Vision and Pattern Recognition

What problem does this paper attempt to address?

The problem that this paper attempts to solve is the current deficiency in the predictive reasoning ability of multimodal large language models (MLLMs). Although these models perform well in perception and interpretation tasks, their ability to perform predictive reasoning based on multiple visual inputs has not been fully explored. To this end, the author proposes a new benchmarking method, aiming to evaluate the predictive reasoning ability of MLLMs in three important areas: abstract pattern reasoning, human activity prediction, and physical interaction prediction. In addition, the author also develops three evaluation methods, using large - language models to quantify the performance of models in predicting and reasoning about future situations based on multiple visual contexts. Experiments prove that the proposed benchmarking and evaluation methods are reasonable and reveal the advantages and disadvantages of currently popular MLLMs in predictive reasoning tasks. This research provides a standardized evaluation framework for the further development of MLLMs, which helps promote the research and development of more advanced models, enabling them to have stronger reasoning and predictive abilities when dealing with complex long - sequence multimodal inputs.