Abstract:Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at <a class="link-external link-https" href="https://github.com/zjwang21/MoE-LPR.git" rel="external noopener nofollow">this https URL</a>.

Lifelong Language Pretraining with Distribution-Specialized Experts

Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

LEMoE: Advanced Mixture of Experts Adaptor for Lifelong Model Editing of Large Language Models

Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapters

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

Lory: Fully Differentiable Mixture-of-Experts for Autoregressive Language Model Pre-training

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

Towards Lifelong Learning of Large Language Models: A Survey

Branch-Train-MiX: Mixing Expert LLMs into a Mixture-of-Experts LLM

ProfLLM: A Framework for Adapting Offline Large Language Models to Few-Shot Expert Knowledge

Leveraging Open Knowledge for Advancing Task Expertise in Large Language Models

MoExtend: Tuning New Experts for Modality and Task Extension

Let the Expert Stick to His Last: Expert-Specialized Fine-Tuning for Sparse Architectural Large Language Models

MoDULA: Mixture of Domain-Specific and Universal LoRA for Multi-Task Learning

AdaMoLE: Fine-Tuning Large Language Models with Adaptive Mixture of Low-Rank Adaptation Experts

MoME: Mixture of Multimodal Experts for Generalist Multimodal Large Language Models

ELDER: Enhancing Lifelong Model Editing with Mixture-of-LoRA

Diversifying the Expert Knowledge for Task-Agnostic Pruning in Sparse Mixture-of-Experts

Enhance Lifelong Model Editing with Continuous Data-Adapter Association

MoRAL: MoE Augmented LoRA for LLMs' Lifelong Learning