Abstract:Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to efficiently scale large language and image-text models, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization. Our findings highlight the substantial potential of MoE frameworks in advancing MLLMs and the code is available at

On Uni-modal Feature Learning in Multi-modal Learning

On Uni-Modal Feature Learning in Supervised Multi-Modal Learning

Improving Multi-Modal Learning with Uni-Modal Teachers

What Makes Multi-modal Learning Better than Single (Provably)

Improving Discriminative Multi-Modal Learning with Large-Scale Pre-Trained Models

Dual Low-Rank Multimodal Fusion

What Makes for Robust Multi-Modal Models in the Face of Missing Modalities?

Dense Multimodal Fusion for Hierarchically Joint Representation

Learn to Combine Modalities in Multimodal Deep Learning

Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably).

Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation

Learning Deep Multimodal Feature Representation with Asymmetric Multi-layer Fusion

Multi-Modal Representation via Contrastive Learning with Attention Bottleneck Fusion and Attentive Statistics Features

Weakly paired multimodal fusion using multilayer extreme learning machine

Supervised Multi-Modal Fission Learning

One-stage Modality Distillation for Incomplete Multimodal Learning

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

Learning Robust Anymodal Segmentor with Unimodal and Cross-modal Distillation

MMTM: Multimodal Transfer Module for CNN Fusion

A cross modal hierarchical fusion multimodal sentiment analysis method based on multi-task learning

Progressive Fusion for Multimodal Integration