Abstract:Recent advancements in large language model (LLM) unlearning have shown remarkable success in removing unwanted data-model influences while preserving the model's utility for legitimate knowledge. However, despite these strides, sparse Mixture-of-Experts (MoE) LLMs--a key subset of the LLM family--have received little attention and remain largely unexplored in the context of unlearning. As MoE LLMs are celebrated for their exceptional performance and highly efficient inference processes, we ask: How can unlearning be performed effectively and efficiently on MoE LLMs? And will traditional unlearning methods be applicable to MoE architectures? Our pilot study shows that the dynamic routing nature of MoE LLMs introduces unique challenges, leading to substantial utility drops when existing unlearning methods are applied. Specifically, unlearning disrupts the router's expert selection, causing significant selection shift from the most unlearning target-related experts to irrelevant ones. As a result, more experts than necessary are affected, leading to excessive forgetting and loss of control over which knowledge is erased. To address this, we propose a novel single-expert unlearning framework, referred to as UOE, for MoE LLMs. Through expert attribution, unlearning is concentrated on the most actively engaged expert for the specified knowledge. Concurrently, an anchor loss is applied to the router to stabilize the active state of this targeted expert, ensuring focused and controlled unlearning that preserves model utility. The proposed UOE framework is also compatible with various unlearning algorithms. Extensive experiments demonstrate that UOE enhances both forget quality up to 5% and model utility by 35% on MoE LLMs across various benchmarks, LLM architectures, while only unlearning 0.06% of the model parameters.

Semi-Supervised Learning of Noisy Mixture of Experts Models

A Survey on Mixture of Experts

Learning Mixtures of Experts with EM

HMoE: Heterogeneous Mixture of Experts for Language Modeling

Distributed Learning of Mixtures of Experts

On Least Square Estimation in Softmax Gating Mixture of Experts

Diversifying the Mixture-of-Experts Representation for Language Models with Orthogonal Optimizer

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

Regularized Maximum Likelihood Estimation and Feature Selection in Mixtures-of-Experts Models

Theory on Mixture-of-Experts in Continual Learning

Unchosen Experts Can Contribute Too: Unleashing MoE Models' Power by Self-Contrast

A Closer Look into Mixture-of-Experts in Large Language Models

Generalization Error Analysis for Sparse Mixture-of-Experts: A Preliminary Study

AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference

MoEC: Mixture of Expert Clusters

Mixture of Diverse Size Experts

OLMoE: Open Mixture-of-Experts Language Models

UOE: Unlearning One Expert Is Enough For Mixture-of-experts LLMS

Omni-SMoLA: Boosting Generalist Multimodal Models with Soft Mixture of Low-rank Experts

Theory of Mixture-of-Experts for Mobile Edge Computing

SEER-MoE: Sparse Expert Efficiency through Regularization for Mixture-of-Experts