Abstract:Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at <a class="link-external link-https" href="https://github.com/zjwang21/MoE-LPR.git" rel="external noopener nofollow">this https URL</a>.

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models Via MoE-Style Plugin.

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

GraphLoRA: Empowering LLMs Fine-Tuning via Graph Collaboration of MoE

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models

ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA

MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting

SwapMoE: Efficient Memory-Constrained Serving of Large Sparse MoE Models Via Dynamic Expert Pruning and Swapping

LocMoE: A Low-Overhead MoE for Large Language Model Training

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models