Abstract:The rapid advancement of large language models (LLMs) has led to architectures with billions to trillions of parameters, posing significant deployment challenges due to their substantial demands on memory, processing power, and energy consumption. Sparse Mixture-of-Experts (SMoE) architectures have emerged as a solution, activating only a subset of parameters per token, thereby achieving faster inference while maintaining performance. However, SMoE models still face limitations in broader deployment due to their large parameter counts and significant GPU memory requirements. In this work, we introduce a gradient-free evolutionary strategy named EEP (Efficient Expert P}runing) to enhance the pruning of experts in SMoE models. EEP relies solely on model inference (i.e., no gradient computation) and achieves greater sparsity while maintaining or even improving performance on downstream tasks. EEP can be used to reduce both the total number of experts (thus saving GPU memory) and the number of active experts (thus accelerating inference). For example, we demonstrate that pruning up to 75% of experts in Mixtral $8\times7$B-Instruct results in a substantial reduction in parameters with minimal performance loss. Remarkably, we observe improved performance on certain tasks, such as a significant increase in accuracy on the SQuAD dataset (from 53.4% to 75.4%), when pruning half of the experts. With these results, EEP not only lowers the barrier to deploying SMoE models,but also challenges the conventional understanding of model pruning by showing that fewer experts can lead to better task-specific performance without any fine-tuning. Code is available at <a class="link-external link-https" href="https://github.com/imagination-research/EEP" rel="external noopener nofollow">this https URL</a>.

Discussion: Effective and Interpretable Outcome Prediction by Training Sparse Mixtures of Linear Experts

Quantifying Explainability in Outcome-Oriented Predictive Process Monitoring

Efficient Expert Pruning for Sparse Mixture-of-Experts Language Models: Enhancing Performance and Reducing Inference Costs

Sparse Expansion and Neuronal Disentanglement

Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models

Fast Deep Mixtures of Gaussian Process Experts

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

Explainability in Process Outcome Prediction: Guidelines to Obtain Interpretable and Faithful Models

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

Parameter-Efficient Sparsity Crafting from Dense to Mixture-of-Experts for Instruction Tuning on General Tasks

Deep interpretable ensembles

Multi-Task Dense Prediction via Mixture of Low-Rank Experts

Sparse Backpropagation for MoE Training

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

IDEA: Interpretable Dynamic Ensemble Architecture for Time Series Prediction

Learning Mixtures of Experts with EM

Deep learning from strongly mixing observations: Sparse-penalized regularization and minimax optimality

Ensemble Interpretation: A Unified Method for Interpretable Machine Learning

DSelect-k: Differentiable Selection in the Mixture of Experts with Applications to Multi-Task Learning

Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning