Abstract:Mixture-of-Experts large language models (MoE-LLMs) marks a significant step forward of language models, however, they encounter two critical challenges in practice: 1) expert parameters lead to considerable memory consumption and loading latency; and 2) the current activated experts are redundant, as many tokens may only require a single expert. Motivated by these issues, we investigate the MoE-LLMs and make two key observations: a) different experts exhibit varying behaviors on activation reconstruction error, routing scores, and activated frequencies, highlighting their differing importance, and b) not all tokens are equally important -- only a small subset is critical. Building on these insights, we propose MC-MoE, a training-free Mixture-Compressor for MoE-LLMs, which leverages the significance of both experts and tokens to achieve an extreme compression. First, to mitigate storage and loading overheads, we introduce Pre-Loading Mixed-Precision Quantization, which formulates the adaptive bit-width allocation as a Linear Programming problem, where the objective function balances multi-factors reflecting the importance of each expert. Additionally, we develop Online Dynamic Pruning, which identifies important tokens to retain and dynamically select activated experts for other tokens during inference to optimize efficiency while maintaining performance. Our MC-MoE integrates static quantization and dynamic pruning to collaboratively achieve extreme compression for MoE-LLMs with less accuracy loss, ensuring an optimal trade-off between performance and efficiency. Extensive experiments confirm the effectiveness of our approach. For instance, at 2.54 bits, MC-MoE compresses 76.6% of the model, with only a 3.8% average accuracy loss. During dynamic inference, we further reduce activated parameters by 15%, with a performance drop of less than 0.6%.

NeuReduce: Reducing Mixed Boolean-Arithmetic Expressions by Recurrent Neural Network

Simplification of General Mixed Boolean-Arithmetic Expressions: GAMBA

Deobfuscation of Semi-Linear Mixed Boolean-Arithmetic Expressions

Towards Interpreting Recurrent Neural Networks Through Probabilistic Abstraction

Simplifying MBA Expression Using E-Graphs

SSLEM: A Simplifier for MBA Expressions based on Semi-linear MBA Expressions and Program Synthesis

Neural Arithmetic Expression Calculator.

Neural Arithmetic Logic Units with Two Transition Matrix and Independent Gates

Syntax-Guided Program Reduction for Understanding Neural Code Intelligence Models

Handwritten Mathematical Expression Recognition via Attention Aggregation Based Bi-directional Mutual Learning

SME: ReRAM-based Sparse-Multiplication-Engine to Squeeze-Out Bit Sparsity of Neural Network

Rethinking Token Reduction for State Space Models

Rethinking Token Reduction in MLLMs: Towards a Unified Paradigm for Training-Free Acceleration

Imbalanced Conditional Conv-Transformer for Mathematical Expression Recognition

Interpreting and Improving Large Language Models in Arithmetic Calculation

Compute Like Humans: Interpretable Step-by-step Symbolic Computation with Deep Neural Network

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More

Efficient and Flexible Method for Reducing Moderate-size Deep Neural Networks with Condensation

Reducing the Need for Backpropagation and Discovering Better Optima With Explicit Optimizations of Neural Networks

BOLD: Boolean Logic Deep Learning