Abstract:Exploiting scale to revamp information absorption has recently become central to the success of deep learning and transformers have become $\textit{de facto}$ choice achieving numerous breakthrough performances on many real-world applications. Despite their enormous success, gigantic transformers suffer not only from exorbitant computational and memory footprints during training but also from severe collapse as evidenced by a high degree of parameter redundancy. Recently proposed Sparsely-activated Mixture-of-Experts (SMoEs) models have shown promise to mitigate the issue of training efficiency, yet they have some critical limitations. In particular, SMoEs models are prone to $\textit{redundant experts}$ due to representational collapse and \textit{poor scalability during inference and downstream fine-tuning} primarily due to overfitting of the learned routing policy to the number of activated experts during training. As recent research efforts are predominantly focused on improving routing policies to encourage expert specializations, our work focuses on $\textit{exploring the overlooked scalability bottleneck of SMoEs}$, to effectively benefit scaling large-scale transformers. To this end, we propose a new plug-and-play training framework, $\textbf{SMoE-Dropout}$ to enable scaling transformers to better accuracy in the full capacity setting without collapse. Specifically, SMoE-Dropout consists of a $\textit{randomly initialized and fixed}$ router network to activate experts and gradually increase their number as training progresses over time. SMoE-Dropout naturally provides a $\textbf{``self-slimmable”}$ property offering consistent boosted performance for transformers with an increase in activated experts during inference and downstream fine-tuning, subjected to resource availability. Our extensive experiments across diverse transformer architectures on a variety of tasks validate superior performance and substantial computation savings, compared to densely trained baselines with equivalent parameter counts. More precisely, our trained BERT outperforms their densely trained counterpart with consistent improvements of {$1.03\%$, $0.78\%$, $1.09\%$} on challenging reasoning tasks {$\texttt{ASDiv-A}$, $\texttt{MAWPS}$, $\texttt{SVAMP}$}, respectively. Codes and models will be publicly released.

Towards More Effective and Economic Sparsely-Activated Model

Exploring Sparse Expert Models and Beyond

Efficient Routing in Sparse Mixture-of-Experts

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs

On the effectiveness of discrete representations in sparse mixture of experts

ExpertFlow: Optimized Expert Activation and Token Allocation for Efficient Mixture-of-Experts Inference

HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation

Towards Being Parameter-Efficient: A Stratified Sparsely Activated Transformer with Dynamic Capacity

On the Representation Collapse of Sparse Mixture of Experts

Mixture of Experts for Network Optimization: A Large Language Model-enabled Approach

XMoE: Sparse Models with Fine-grained and Adaptive Expert Selection

Harder Tasks Need More Experts: Dynamic Routing in MoE Models

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

Sparse MoE with Random Routing as the New Dropout: Training Bigger and Self-Scalable Models

ME-Switch: A Memory-Efficient Expert Switching Framework for Large Language Models

Taming Sparsely Activated Transformer with Stochastic Experts

Janus: A Unified Distributed Training Framework for Sparse Mixture-of-Experts Models

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts