Abstract:The Mixture of Experts (MoE) has emerged as a highly successful technique in deep learning, based on the principle of divide-and-conquer to maximize model capacity without significant additional computational cost. Even in the era of large-scale language models (LLMs), MoE continues to play a crucial role, as some researchers have indicated that GPT-4 adopts the MoE structure to ensure diverse inference results. However, MoE is susceptible to performance degeneracy, particularly evident in the issues of imbalance and homogeneous representation among experts. While previous studies have extensively addressed the problem of imbalance, the challenge of homogeneous representation remains unresolved. In this study, we shed light on the homogeneous representation problem, wherein experts in the MoE fail to specialize and lack diversity, leading to frustratingly high similarities in their representations (up to 99\% in a well-performed MoE model). This problem restricts the expressive power of the MoE and, we argue, contradicts its original intention. To tackle this issue, we propose a straightforward yet highly effective solution: OMoE, an orthogonal expert optimizer. Additionally, we introduce an alternating training strategy that encourages each expert to update in a direction orthogonal to the subspace spanned by other experts. Our algorithm facilitates MoE training in two key ways: firstly, it explicitly enhances representation diversity, and secondly, it implicitly fosters interaction between experts during orthogonal weights computation. Through extensive experiments, we demonstrate that our proposed optimization algorithm significantly improves the performance of fine-tuning the MoE model on the GLUE benchmark, SuperGLUE benchmark, question-answering task, and name entity recognition tasks.

Enhancing diversity for logical table‐to‐text generation with mixture of experts

TabMoE: A General Framework for Diverse Table-Based Reasoning with Mixture-of-Experts

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer

Three-stage Logical Table-to-Text Generation Based on Type Control

Improving Logical-Level Natural Language Generation with Topic-Conditioned Data Augmentation and Logical Form Generation

HMoE: Heterogeneous Mixture of Experts for Language Modeling

LOGEN: Few-shot Logical Knowledge-Conditioned Text Generation with Self-training.

MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

Mixture of Diverse Size Experts

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

Merging Experts into One: Improving Computational Efficiency of Mixture of Experts

Sweeping Heterogeneity with Smart MoPs: Mixture of Prompts for LLM Task Adaptation

Improving Linguistic Diversity of Large Language Models with Possibility Exploration Fine-Tuning

A Survey on Mixture of Experts

Dynamic Data Mixing Maximizes Instruction Tuning for Mixture-of-Experts

Screening Through a Broad Pool: Towards Better Diversity for Lexically Constrained Text Generation

A Closer Look into Mixture-of-Experts in Large Language Models

Investigating the Robustness of Natural Language Generation from Logical Forms Via Counterfactual Samples

Improving Structural Diversity of Blackbox LLMs via Chain-of-Specification Prompting

Graph Mixture of Experts: Learning on Large-Scale Graphs with Explicit Diversity Modeling

Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast