Abstract:While multimodal large language models (MLLMs) have demonstrated extraordinary vision-language understanding capabilities and shown potential to serve as general-purpose assistants, their abilities to solve instance-level visual-language problems beyond a single image warrant further exploration. In order to assess these unproven abilities of MLLMs, this paper proposes a new visual grounding task called multi-context visual grounding, which aims to localize instances of interest across multiple images based on open-ended text prompts. To facilitate this research, we meticulously construct a new dataset MC-Bench for benchmarking the visual grounding capabilities of MLLMs. MC-Bench features 2K high-quality and manually annotated samples, consisting of instance-level labeled image pairs and corresponding text prompts that indicate the target instances in the images. In total, there are three distinct styles of text prompts, covering 20 practical skills. We benchmark over 20 state-of-the-art MLLMs and foundation models with potential multi-context visual grounding capabilities. Our evaluation reveals a non-trivial performance gap between existing MLLMs and humans across all metrics. We also observe that existing MLLMs typically outperform foundation models without LLMs only on image-level metrics, and the specialist MLLMs trained on single images often struggle to generalize to multi-image scenarios. Moreover, a simple stepwise baseline integrating advanced MLLM and a detector can significantly surpass prior end-to-end MLLMs. We hope our MC-Bench and empirical findings can encourage the research community to further explore and enhance the untapped potentials of MLLMs in instance-level tasks, particularly in multi-image contexts. Project page: <a class="link-external link-https" href="https://xuyunqiu.github.io/MC-Bench/" rel="external noopener nofollow">this https URL</a>.

Can MLLMs Perform Text-to-Image In-Context Learning?

VL-ICL Bench: The Devil in the Details of Multimodal In-Context Learning

What Makes Multimodal In-Context Learning Work?

CODIS: Benchmarking Context-Dependent Visual Comprehension for Multimodal Large Language Models

TextCoT: Zoom In for Enhanced Multimodal Text-Rich Image Understanding

Multimodal Contrastive In-Context Learning

MIBench: Evaluating Multimodal Large Language Models over Multiple Images

Improving Context Understanding in Multimodal Large Language Models Via Multimodal Composition Learning

CoCoT: Contrastive Chain-of-Thought Prompting for Large Multimodal Models with Multiple Image Inputs

CoMM: A Coherent Interleaved Image-Text Dataset for Multimodal Understanding and Generation

MCTBench: Multimodal Cognition towards Text-Rich Visual Scenes Benchmark

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

MMICT: Boosting Multi-Modal Fine-Tuning with In-Context Examples

Towards Multimodal In-Context Learning for Vision & Language Models

MC-Bench: A Benchmark for Multi-Context Visual Grounding in the Era of MLLMs

ConTextual: Evaluating Context-Sensitive Text-Rich Visual Reasoning in Large Multimodal Models

Link-Context Learning for Multimodal LLMs

MileBench: Benchmarking MLLMs in Long Context

LAMM: Language-Assisted Multi-Modal Instruction-Tuning Dataset, Framework, and Benchmark

Visual CoT: Advancing Multi-Modal Language Models with a Comprehensive Dataset and Benchmark for Chain-of-Thought Reasoning

Towards More Unified In-context Visual Understanding