Abstract:The rapid advancement of Large Language Models (LLMs) has led to an influx of efforts to extend their capabilities to multimodal tasks. Among them, growing attention has been focused on monolithic Multimodal Large Language Models (MLLMs) that integrate visual encoding and language decoding into a single LLM. Despite the structural simplicity and deployment-friendliness, training a monolithic MLLM with promising performance still remains challenging. In particular, the popular approaches adopt continuous pre-training to extend a pre-trained LLM to a monolithic MLLM, which suffers from catastrophic forgetting and leads to performance degeneration. In this paper, we aim to overcome this limitation from the perspective of delta tuning. Specifically, our core idea is to embed visual parameters into a pre-trained LLM, thereby incrementally learning visual knowledge from massive data via delta tuning, i.e., freezing the LLM when optimizing the visual parameters. Based on this principle, we present Mono-InternVL, a novel monolithic MLLM that seamlessly integrates a set of visual experts via a multimodal mixture-of-experts structure. Moreover, we propose an innovative pre-training strategy to maximize the visual capability of Mono-InternVL, namely Endogenous Visual Pre-training (EViP). In particular, EViP is designed as a progressive learning process for visual experts, which aims to fully exploit the visual knowledge from noisy data to high-quality data. To validate our approach, we conduct extensive experiments on 16 benchmarks. Experimental results not only validate the superior performance of Mono-InternVL compared to the state-of-the-art MLLM on 6 multimodal benchmarks, e.g., +113 points over InternVL-1.5 on OCRBench, but also confirm its better deployment efficiency, with first token latency reduced by up to 67%.

Demonstrative Instruction Following in Multimodal LLMs Via Integrating Low-Rank Adaptation with Ensemble Learning

InfMLLM: A Unified Framework for Visual-Language Tasks.

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

NoteLLM-2: Multimodal Large Representation Models for Recommendation

EMMA: Efficient Visual Alignment in Multi-Modal LLMs

Fine-tuning Multimodal LLMs to Follow Zero-shot Demonstrative Instructions

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

Feast Your Eyes: Mixture-of-Resolution Adaptation for Multimodal Large Language Models

Unified Generative and Discriminative Training for Multi-modal Large Language Models

MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning

AIM: Let Any Multi-modal Large Language Models Embrace Efficient In-Context Learning

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Browse and Concentrate: Comprehending Multimodal Content via prior-LLM Context Fusion

Multi-modal Instruction Tuned LLMs with Fine-grained Visual Perception

LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledge

OneLLM: One Framework to Align All Modalities with Language

MA-LMM: Memory-Augmented Large Multimodal Model for Long-Term Video Understanding

Mono-InternVL: Pushing the Boundaries of Monolithic Multimodal Large Language Models with Endogenous Visual Pre-training

MM1: Methods, Analysis & Insights from Multimodal LLM Pre-training

Visual Cue Enhancement and Dual Low-Rank Adaptation for Efficient Visual Instruction Fine-Tuning

Position-Enhanced Visual Instruction Tuning for Multimodal Large Language Models