Abstract:Recent advancements in Multimodal Large Language Models (MLLMs) underscore the significance of scalable models and data to boost performance, yet this often incurs substantial computational costs. Although the Mixture of Experts (MoE) architecture has been employed to efficiently scale large language and image-text models, these efforts typically involve fewer experts and limited modalities. To address this, our work presents the pioneering attempt to develop a unified MLLM with the MoE architecture, named Uni-MoE that can handle a wide array of modalities. Specifically, it features modality-specific encoders with connectors for a unified multimodal representation. We also implement a sparse MoE architecture within the LLMs to enable efficient training and inference through modality-level data parallelism and expert-level model parallelism. To enhance the multi-expert collaboration and generalization, we present a progressive training strategy: 1) Cross-modality alignment using various connectors with different cross-modality data, 2) Training modality-specific experts with cross-modality instruction data to activate experts' preferences, and 3) Tuning the Uni-MoE framework utilizing Low-Rank Adaptation (LoRA) on mixed multimodal instruction data. We evaluate the instruction-tuned Uni-MoE on a comprehensive set of multimodal datasets. The extensive experimental results demonstrate Uni-MoE's principal advantage of significantly reducing performance bias in handling mixed multimodal datasets, alongside improved multi-expert collaboration and generalization. Our findings highlight the substantial potential of MoE frameworks in advancing MLLMs and the code is available at

Enhanced Experts with Uncertainty-Aware Routing for Multimodal Sentiment Analysis

MMoE: Enhancing Multimodal Models with Mixtures of Multimodal Interaction Experts

Embracing Unimodal Aleatoric Uncertainty for Robust Multimodal Fusion

Expert-Token Resonance: Redefining MoE Routing through Affinity-Driven Active Selection

Hybrid Uncertainty Calibration for Multimodal Sentiment Analysis

Efficient Deweather Mixture-of-Experts with Uncertainty-aware Feature-wise Linear Modulation

MaskMoE: Boosting Token-Level Learning via Routing Mask in Mixture-of-Experts

Uncertainty-Debiased Multimodal Fusion: Learning Deterministic Joint Representation for Multimodal Sentiment Analysis

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts

Training of Neural Networks with Uncertain Data: A Mixture of Experts Approach

HMoE: Heterogeneous Mixture of Experts for Language Modeling

Modeling Hierarchical Uncertainty for Multimodal Emotion Recognition in Conversation

GW-MoE: Resolving Uncertainty in MoE Router with Global Workspace Theory

Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models

UniMSE: Towards Unified Multimodal Sentiment Analysis and Emotion Recognition

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

Harder Tasks Need More Experts: Dynamic Routing in MoE Models

CartesianMoE: Boosting Knowledge Sharing among Experts via Cartesian Product Routing in Mixture-of-Experts

Efficient Routing in Sparse Mixture-of-Experts

Feature Extraction Network with Attention Mechanism for Data Enhancement and Recombination Fusion for Multimodal Sentiment Analysis

AdaMoE: Token-Adaptive Routing with Null Experts for Mixture-of-Experts Language Models