Abstract:In recent years, multimodal large language models (MLLMs) have shown remarkable capabilities in tasks like visual question answering and common sense reasoning, while visual perception models have made significant strides in perception tasks, such as detection and segmentation. However, MLLMs mainly focus on high-level image-text interpretations and struggle with fine-grained visual understanding, and vision perception models usually suffer from open-world distribution shifts due to their limited model capacity. To overcome these challenges, we propose the Mutually Reinforced Multimodal Large Language Model (MR-MLLM), a novel framework that synergistically enhances visual perception and multimodal comprehension. First, a shared query fusion mechanism is proposed to harmonize detailed visual inputs from vision models with the linguistic depth of language models, enhancing multimodal comprehension and vision perception synergistically. Second, we propose the perception-enhanced cross-modal integration method, incorporating novel modalities from vision perception outputs, like object detection bounding boxes, to capture subtle visual elements, thus enriching the understanding of both visual and textual data. In addition, an innovative perception-embedded prompt generation mechanism is proposed to embed perceptual information into the language model's prompts, aligning the responses contextually and perceptually for a more accurate multimodal interpretation. Extensive experiments demonstrate MR-MLLM's superior performance in various multimodal comprehension and vision perception tasks, particularly those requiring corner case vision perception and fine-grained language comprehension.

What problem does this paper attempt to address?

The paper aims to address the limitations of Multimodal Large Language Models (MLLM) in processing visual and textual modality information, particularly the inadequacies in fine-grained visual understanding and the limited generalization ability of visual perception models under open-world distribution changes. To tackle these issues, the authors propose a new framework called the "Mutually Reinforced Multimodal Large Language Model" (MR-MLLM). This framework enhances multimodal understanding and visual perception through the following aspects: 1. **Shared Query Fusion Mechanism**: A mechanism is designed to coordinate detailed visual inputs from the visual model with the linguistic depth of the language model, thereby synergistically improving multimodal understanding and visual perception capabilities. 2. **Perception-Enhanced Cross-Modal Integration Method**: A method is proposed to incorporate new modalities (such as object detection bounding boxes) from visual perception outputs into the model to capture subtle visual elements, thereby enriching the understanding of visual and textual data. 3. **Perception-Embedded Prompt Generation Mechanism**: An innovative approach is developed to embed perceptual information into the prompts of the language model, making responses more consistent in context and perception, thus achieving more accurate multimodal interpretation. Through extensive experiments, the paper demonstrates the superior performance of MR-MLLM in various multimodal understanding and visual perception tasks, especially in tasks requiring corner-case visual perception and fine-grained language understanding. Additionally, MR-MLLM significantly improves the performance of visual perception models in handling corner-case detection tasks and endows closed-set trained professional detectors with the ability to detect new categories.

MR-MLLM: Mutual Reinforcement of Multimodal Comprehension and Vision Perception

VisionLLM v2: An End-to-End Generalist Multimodal Large Language Model for Hundreds of Vision-Language Tasks

Towards Vision Enhancing LLMs: Empowering Multimodal Knowledge Storage and Sharing in LLMs

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

InfMLLM: A Unified Framework for Visual-Language Tasks.

MMICL: Empowering Vision-language Model with Multi-Modal In-Context Learning

Unified Generative and Discriminative Training for Multi-modal Large Language Models

A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Tasks

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

Efficient Multimodal Large Language Models: A Survey

X-Former: Unifying Contrastive and Reconstruction Learning for MLLMs

A Survey on Evaluation of Multimodal Large Language Models

LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge

A Comprehensive Review of Multimodal Large Language Models: Performance and Challenges Across Different Tasks

Enhancing Perception Capabilities of Multimodal LLMs with Training-free Fusion

ActiView: Evaluating Active Perception Ability for Multimodal Large Language Models

A Survey on Benchmarks of Multimodal Large Language Models

Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion

Enhancing Multimodal Large Language Models with Multi-instance Visual Prompt Generator for Visual Representation Enrichment

A Survey on Multimodal Large Language Models