Abstract:Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to efficiently scale large language and image-text models, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization. Our findings highlight the substantial potential of MoE frameworks in advancing MLLMs and the code is available at

What problem does this paper attempt to address?

This paper primarily discusses how to use the Mixture of Experts (MoE) architecture to extend Multimodal Large Language Models (MLLMs) for handling various modalities of data such as images, texts, audios, and videos, thus improving efficiency and performance. Existing multimodal models often face high computational costs in terms of scale and data expansion. To address this problem, the paper proposes a new model called Uni-MoE, which adopts a sparse MoE architecture that allows efficient training and inference through modality-level data parallelism and expert-level model parallelism. The characteristics of Uni-MoE include: 1. Modality-specific encoders: used to obtain the encodings of different modalities and map them into a unified language representation space through connectors. 2. Sparse MoE layer: introduced within the dense LSTM blocks, each MoE block contains a shared self-attention layer across all modalities, diversity experts based on FFN, and a sparse router for token-level knowledge allocation. To enhance collaboration and generalization among multiple experts, the paper proposes a three-stage progressive training strategy: 1. Cross-modal alignment: train the respective connectors using various connectors and cross-modal data. 2. Training modality-specific experts: train experts for each modality using cross-modal datasets to activate their expertise. 3. Fine-tuning the Uni-MoE framework: fine-tune the mixed multimodal instruction data using LoRA technology to further reduce training costs. Experimental results show that Uni-MoE significantly reduces performance bias when dealing with mixed multimodal datasets, and improves collaboration and generalization among multiple experts. Additionally, compared to traditional dense models, the MoE framework performs well in handling complex tasks such as videos and long speeches, while also enhancing stability and robustness across different modalities. The paper also found that integrating more modal information into Uni-MoE can enhance performance in single-modality tasks. In conclusion, this paper aims to solve the computational efficiency problem faced by Multimodal Large Language Models in scaling and handling multimodal data through the MoE architecture. Through innovative training strategies and model design, efficient and unified multimodal understanding is achieved.

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

Uni-Med: A Unified Medical Generalist Foundation Model For Multi-Task Learning Via Connector-MoE

CuMo: Scaling Multimodal LLM with Co-Upcycled Mixture-of-Experts

MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More

MoExtend: Tuning New Experts for Modality and Task Extension

Med-MoE: Mixture of Domain-Specific Experts for Lightweight Medical Vision-Language Models

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

HMoE: Heterogeneous Mixture of Experts for Language Modeling

UnifiedMLLM: Enabling Unified Representation for Multi-modal Multi-tasks With Large Language Model

LocMoE: A Low-Overhead MoE for Large Language Model Training

Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts

EE-MLLM: A Data-Efficient and Compute-Efficient Multimodal Large Language Model

A Survey on Mixture of Experts

AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

A Closer Look into Mixture-of-Experts in Large Language Models

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

MoVA: Adapting Mixture of Vision Experts to Multimodal Context

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free