Efficiently Democratizing Medical LLMs for 50 Languages via a Mixture of Language Family Experts

Guorui Zheng,Xidong Wang,Juhao Liang,Nuo Chen,Yuping Zheng,Benyou Wang
2024-10-14
Abstract:Adapting medical Large Language Models to local languages can reduce barriers to accessing healthcare services, but data scarcity remains a significant challenge, particularly for low-resource languages. To address this, we first construct a high-quality medical dataset and conduct analysis to ensure its quality. In order to leverage the generalization capability of multilingual LLMs to efficiently scale to more resource-constrained languages, we explore the internal information flow of LLMs from a multilingual perspective using Mixture of Experts (MoE) modularity. Technically, we propose a novel MoE routing method that employs language-specific experts and cross-lingual routing. Inspired by circuit theory, our routing analysis revealed a Spread Out in the End information flow mechanism: while earlier layers concentrate cross-lingual information flow, the later layers exhibit language-specific divergence. This insight directly led to the development of the Post-MoE architecture, which applies sparse routing only in the later layers while maintaining dense others. Experimental results demonstrate that this approach enhances the generalization of multilingual models to other languages while preserving interpretability. Finally, to efficiently scale the model to 50 languages, we introduce the concept of language family experts, drawing on linguistic priors, which enables scaling the number of languages without adding additional parameters.
Computation and Language
What problem does this paper attempt to address?
### Problems Addressed by the Paper This paper aims to address the adaptability of large language models (LLMs) in the medical field across multiple languages, especially for resource-poor languages. Specifically, the paper focuses on the following key issues: 1. **Data Scarcity**: - The lack of high-quality multilingual data in the medical field limits the development and application of models in low-resource languages. Although some research has made progress in this area, these studies usually focus on only a few major languages, typically fewer than 6. 2. **Model Generalization**: - How to leverage the generalization capabilities of multilingual LLMs to effectively extend to more resource-poor languages is a significant challenge. Existing methods often struggle to balance model performance and efficiency. 3. **Interpretability of Multilingual Extension**: - In a multilingual environment, how to explain the internal information flow mechanism of the model to ensure its transparency and interpretability is also an issue that needs to be addressed. ### Solutions To address the above issues, the paper proposes the following methods: 1. **Building a High-Quality Multilingual Medical Dataset**: - The authors first constructed a high-quality medical dataset covering 12 major languages and conducted rigorous quality assessments. This step provides a solid foundation for subsequent model training. 2. **Mixture of Experts (MoE)**: - By introducing a mixture of experts model, particularly language-specific experts and cross-language routing mechanisms, the paper proposes a new routing method (Hybrid-k). This method not only activates language-specific experts but also transfers general knowledge between different languages. 3. **Information Flow Circuit Analysis**: - Using circuit theory, the authors analyzed the internal information flow mechanism of the model and discovered a phenomenon of "final dispersion": early layers handle cross-language information centrally, while later layers exhibit language-specific differentiation. This finding provides theoretical support for designing more effective model architectures. 4. **Post-MoE Architecture**: - Based on the "final dispersion" phenomenon, the paper proposes the Post-MoE architecture, which applies sparse routing only in the later layers of the model, thereby improving the model's generalization ability while maintaining performance. 5. **Language Family Experts**: - To further extend the model's multilingual capabilities, the authors introduced the concept of language family experts, classifying 50 languages into 7 language families. This approach not only reduces the number of parameters but also enhances the model's scalability and robustness. ### Experimental Results Experimental results show that the proposed Post-MoE architecture outperforms existing dense models and other multilingual models across multiple languages. The performance improvement is particularly significant in resource-poor languages. Additionally, through the language family experts approach, the model can effectively extend to 50 languages without requiring additional parameters. ### Main Contributions 1. Constructed a high-quality multilingual medical dataset covering 12 major languages and validated its quality through experiments. 2. Proposed a new method based on circuit theory to explain the routing mechanism in a multilingual environment, discovering the "final dispersion" phenomenon. 3. Efficiently extended the medical LLM to 50 languages by introducing language family experts, demonstrating its potential for further language expansion.