Abstract:Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at <a class="link-external link-https" href="https://github.com/zjwang21/MoE-LPR.git" rel="external noopener nofollow">this https URL</a>.

HDMoLE: Mixture of LoRA Experts with Hierarchical Routing and Dynamic Thresholds for Fine-Tuning LLM-based ASR Models

AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts

Mixture-of-LoRAs: An Efficient Multitask Tuning for Large Language Models

Mixture of LoRA Experts

SLIM: Let LLM Learn More and Forget Less with Soft LoRA and Identity Mixture

LLaVA-MoLE: Sparse Mixture of LoRA Experts for Mitigating Data Conflicts in Instruction Finetuning MLLMs

MiLoRA: Efficient Mixture of Low-Rank Adaptation for Large Language Models Fine-tuning

MixLoRA: Enhancing Large Language Models Fine-Tuning with LoRA-based Mixture of Experts

AlphaLoRA: Assigning LoRA Experts Based on Layer Training Quality

Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM

When MOE Meets LLMs: Parameter Efficient Fine-tuning for Multi-task Medical Applications

MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning

MoELoRA: Contrastive Learning Guided Mixture of Experts on Parameter-Efficient Fine-Tuning for Large Language Models

MALoRA: Mixture of Asymmetric Low-Rank Adaptation for Enhanced Multi-Task Learning

SAML: Speaker Adaptive Mixture of LoRA Experts for End-to-End ASR

Language-Routing Mixture of Experts for Multilingual and Code-Switching Speech Recognition

Enhancing Code-Switching Speech Recognition with LID-Based Collaborative Mixture of Experts Model

U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF

A Framework to Implement 1+N Multi-task Fine-tuning Pattern in LLMs Using the CGC-LORA Algorithm

MoDE: Effective Multi-task Parameter Efficient Fine-Tuning with a Mixture of Dyadic Experts

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing