Abstract:Recent advancements in large language model (LLM) unlearning have shown remarkable success in removing unwanted data-model influences while preserving the model's utility for legitimate knowledge. However, despite these strides, sparse Mixture-of-Experts (MoE) LLMs--a key subset of the LLM family--have received little attention and remain largely unexplored in the context of unlearning. As MoE LLMs are celebrated for their exceptional performance and highly efficient inference processes, we ask: How can unlearning be performed effectively and efficiently on MoE LLMs? And will traditional unlearning methods be applicable to MoE architectures? Our pilot study shows that the dynamic routing nature of MoE LLMs introduces unique challenges, leading to substantial utility drops when existing unlearning methods are applied. Specifically, unlearning disrupts the router's expert selection, causing significant selection shift from the most unlearning target-related experts to irrelevant ones. As a result, more experts than necessary are affected, leading to excessive forgetting and loss of control over which knowledge is erased. To address this, we propose a novel single-expert unlearning framework, referred to as UOE, for MoE LLMs. Through expert attribution, unlearning is concentrated on the most actively engaged expert for the specified knowledge. Concurrently, an anchor loss is applied to the router to stabilize the active state of this targeted expert, ensuring focused and controlled unlearning that preserves model utility. The proposed UOE framework is also compatible with various unlearning algorithms. Extensive experiments demonstrate that UOE enhances both forget quality up to 5% and model utility by 35% on MoE LLMs across various benchmarks, LLM architectures, while only unlearning 0.06% of the model parameters.

Theory on Mixture-of-Experts in Continual Learning

Mixture-of-Variational-Experts for Continual Learning

Theory of Mixture-of-Experts for Mobile Edge Computing

Low-Rank Mixture-of-Experts for Continual Medical Image Segmentation

Mixture of Experts in a Mixture of RL settings

Orchestrate Latent Expertise: Advancing Online Continual Learning with Multi-Level Supervision and Reverse Self-Distillation

UOE: Unlearning One Expert Is Enough For Mixture-of-experts LLMS

Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

Sparsely Activated Mixture-of-Experts are Robust Multi-Task Learners

MoEC: Mixture of Expert Clusters

A Closer Look into Mixture-of-Experts in Large Language Models

A Survey on Mixture of Experts

Semi-Supervised Learning of Noisy Mixture of Experts Models

HMoE: Heterogeneous Mixture of Experts for Language Modeling

Mixture of Experts for Network Optimization: A Large Language Model-enabled Approach

PMoE: Progressive Mixture of Experts with Asymmetric Transformer for Continual Learning

Mixture of Experts for Intelligent Networks: A Large Language Model-enabled Approach

Learning an evolved mixture model for task-free continual learning

Order parameters and phase transitions of continual learning in deep neural networks

Multilinear Mixture of Experts: Scalable Expert Specialization through Factorization