Abstract:Large Language Models (LLMs) are often English-centric due to the disproportionate distribution of languages in their pre-training data. Enhancing non-English language capabilities through post-pretraining often results in catastrophic forgetting of the ability of original languages. Previous methods either achieve good expansion with severe forgetting or slight forgetting with poor expansion, indicating the challenge of balancing language expansion while preventing forgetting. In this paper, we propose a method called MoE-LPR (Mixture-of-Experts with Language Priors Routing) to alleviate this problem. MoE-LPR employs a two-stage training approach to enhance the multilingual capability. First, the model is post-pretrained into a Mixture-of-Experts (MoE) architecture by upcycling, where all the original parameters are frozen and new experts are added. In this stage, we focus improving the ability on expanded languages, without using any original language data. Then, the model reviews the knowledge of the original languages with replay data amounting to less than 1% of post-pretraining, where we incorporate language priors routing to better recover the abilities of the original languages. Evaluations on multiple benchmarks show that MoE-LPR outperforms other post-pretraining methods. Freezing original parameters preserves original language knowledge while adding new experts preserves the learning ability. Reviewing with LPR enables effective utilization of multilingual knowledge within the parameters. Additionally, the MoE architecture maintains the same inference overhead while increasing total model parameters. Extensive experiments demonstrate MoE-LPR's effectiveness in improving expanded languages and preserving original language proficiency with superior scalability. Code and scripts are freely available at <a class="link-external link-https" href="https://github.com/zjwang21/MoE-LPR.git" rel="external noopener nofollow">this https URL</a>.

Mixture of In-Context Experts Enhance LLMs' Long Context Awareness

Your Mixture-of-Experts LLM Is Secretly an Embedding Model For Free

LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens

AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach

Mixture of Experts for Intelligent Networks: A Large Language Model-enabled Approach

Extending LLMs' Context Window with 100 Samples

Octavius: Mitigating Task Interference in MLLMs via LoRA-MoE

Found in the Middle: How Language Models Use Long Contexts Better via Plug-and-Play Positional Encoding

Why Does the Effective Context Length of LLMs Fall Short?

Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models

Mixture of Experts for Network Optimization: A Large Language Model-enabled Approach

Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design

A Closer Look into Mixture-of-Experts in Large Language Models

Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning

Understanding the RoPE Extensions of Long-Context LLMs: An Attention Perspective

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More

HMoE: Heterogeneous Mixture of Experts for Language Modeling

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

Boosting Code-Switching ASR with Mixture of Experts Enhanced Speech-Conditioned LLM

AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference

Expert-Token Resonance: Redefining MoE Routing through Affinity-Driven Active Selection