Abstract:Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at <a class="link-external link-https" href="https://github.com/zjwang21/MoE-LPR.git" rel="external noopener nofollow">this https URL</a>.

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models Via MoE-Style Plugin.

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

Retrieval-Augmented Mixture of LoRA Experts for Uploadable Machine Learning

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

GraphLoRA: Empowering LLMs Fine-Tuning via Graph Collaboration of MoE

AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

MoE-CT: A Novel Approach For Large Language Models Training With Resistance To Catastrophic Forgetting

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models

SwapMoE: Efficient Memory-Constrained Serving of Large Sparse MoE Models Via Dynamic Expert Pruning and Swapping

ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA

DLP-LoRA: Efficient Task-Specific LoRA Fusion with a Dynamic, Lightweight Plugin for Large Language Models

MoE-LLaVA: Mixture of Experts for Large Vision-Language Models

OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning