Abstract:Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to efficiently scale large language and image-text models, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization. Our findings highlight the substantial potential of MoE frameworks in advancing MLLMs and the code is available at

A Unified Visual Prompt Tuning Framework with Mixture-of-Experts for Multimodal Information Extraction.

MuDPT: Multi-modal Deep-symphysis Prompt Tuning for Large Pre-trained Vision-Language Models

MoPE: Parameter-Efficient and Scalable Multimodal Fusion via Mixture of Prompt Experts

UMIE: Unified Multimodal Information Extraction with Instruction Tuning

Mixture-of-Prompt-Experts for Multi-modal Semantic Understanding

UMP: Unified Modality-aware Prompt Tuning for Text-Video Retrieval

Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model

Tuning Multi-mode Token-level Prompt Alignment across Modalities

MoExtend: Tuning New Experts for Modality and Task Extension

Multi-modal Intent Detection with LVAMoE: the Language-Visual-Audio Mixture of Experts

Recognizing Everything from All Modalities at Once: Grounded Multimodal Universal Information Extraction

Scene-Aware Prompt for Multi-modal Dialogue Understanding and Generation

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

Unified Vision and Language Prompt Learning

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts

Conditional Prompt Tuning for Multimodal Fusion

Incorporating Visual Experts to Resolve the Information Loss in Multimodal Large Language Models

Multimedia Event Extraction from News with a Unified Contrastive Learning Framework

Rethinking the constraints of multimodal fusion: case study in Weakly-Supervised Audio-Visual Video Parsing

A Sentimental Prompt Framework with Visual Text Encoder for Multimodal Sentiment Analysis

MePT: Multi-Representation Guided Prompt Tuning for Vision-Language Model