Abstract:Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at <a class="link-external link-https" href="https://github.com/zjwang21/MoE-LPR.git" rel="external noopener nofollow">this https URL</a>.

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

OwLore: Outlier-weighed Layerwise Sampled Low-Rank Projection for Memory-Efficient LLM Fine-tuning

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

LoRAMoE: Alleviate World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models Via MoE-Style Plugin.

LoRAMoE: Alleviating World Knowledge Forgetting in Large Language Models via MoE-Style Plugin

Analyzing and Reducing Catastrophic Forgetting in Parameter Efficient Tuning

PeriodicLoRA: Breaking the Low-Rank Bottleneck in LoRA Optimization

MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuning

MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning

Mixture of LoRA Experts

MoS: Unleashing Parameter Efficiency of Low-Rank Adaptation with Mixture of Shards

Mitigating Forgetting in LLM Supervised Fine-Tuning and Preference Learning

Interweaving Memories of a Siamese Large Language Model

Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

Mixture of In-Context Experts Enhance LLMs' Long Context Awareness