Abstract:While multimodal large language models (MLLMs) have demonstrated extraordinary vision-language understanding capabilities and shown potential to serve as general-purpose assistants, their abilities to solve instance-level visual-language problems beyond a single image warrant further exploration. In order to assess these unproven abilities of MLLMs, this paper proposes a new visual grounding task called multi-context visual grounding, which aims to localize instances of interest across multiple images based on open-ended text prompts. To facilitate this research, we meticulously construct a new dataset MC-Bench for benchmarking the visual grounding capabilities of MLLMs. MC-Bench features 2K high-quality and manually annotated samples, consisting of instance-level labeled image pairs and corresponding text prompts that indicate the target instances in the images. In total, there are three distinct styles of text prompts, covering 20 practical skills. We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities. Our evaluation reveals a non-trivial performance gap between existing MLLMs and humans across all metrics. We also observe that existing MLLMs typically outperform foundation models without LLMs only on image-level metrics, and the specialist MLLMs trained on single images often struggle to generalize to multi-image scenarios. Moreover, a simple stepwise baseline integrating advanced MLLM and a detector can significantly surpass prior end-to-end MLLMs. We hope our MC-Bench and empirical findings can encourage the research community to further explore and enhance the untapped potentials of MLLMs in instance-level tasks, particularly in multi-image contexts. Project page: <a class="link-external link-https" href="https://xuyunqiu.github.io/MC-Bench/" rel="external noopener nofollow">this https URL</a>.

Context-aware chatbot using MLLMs for Cultural Heritage

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Visual Question Answering for Cultural Heritage

Augmenting Multimodal LLMs with Self-Reflective Tokens for Knowledge-based Visual Question Answering

Improving Context Understanding in Multimodal Large Language Models Via Multimodal Composition Learning

Enhanced Multimodal RAG-LLM for Accurate Visual Question Answering

LLMRA: Multi-modal Large Language Model based Restoration Assistant

LLaVA-Docent: Instruction Tuning with Multimodal Large Language Model to Support Art Appreciation Education

Towards Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs

MR-MLLM: Mutual Reinforcement of Multimodal Comprehension and Vision Perception

LOVA3: Learning to Visual Question Answering, Asking and Assessment

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

LLMGA: Multimodal Large Language Model based Generation Assistant

MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning

Diffusion Based Augmentation for Captioning and Retrieval in Cultural Heritage

Understanding Information Storage and Transfer in Multi-modal Large Language Models

The Revolution of Multimodal Large Language Models: A Survey

Empowering MultiModal Models' In-Context Learning Ability through Large Language Models.

InfMLLM: A Unified Framework for Visual-Language Tasks.

An Interactive Multi-modal Query Answering System with Retrieval-Augmented Large Language Models