DeepSeekMoE: Towards Ultimate Expert Specialization in Mixture-of-Experts Language Models

Damai Dai,Chengqi Deng,Chenggang Zhao,R.X. Xu,Huazuo Gao,Deli Chen,Jiashi Li,Wangding Zeng,Xingkai Yu,Y. Wu,Zhenda Xie,Y.K. Li,Panpan Huang,Fuli Luo,Chong Ruan,Zhifang Sui,Wenfeng Liang

2024-01-12

Abstract:In the era of large language models, Mixture-of-Experts (MoE) is a promising architecture for managing computational costs when scaling up model parameters. However, conventional MoE architectures like GShard, which activate the top-$K$ out of $N$ experts, face challenges in ensuring expert specialization, i.e. each expert acquires non-overlapping and focused knowledge. In response, we propose the DeepSeekMoE architecture towards ultimate expert specialization. It involves two principal strategies: (1) finely segmenting the experts into $mN$ ones and activating $mK$ from them, allowing for a more flexible combination of activated experts; (2) isolating $K_s$ experts as shared ones, aiming at capturing common knowledge and mitigating redundancy in routed experts. Starting from a modest scale with 2B parameters, we demonstrate that DeepSeekMoE 2B achieves comparable performance with GShard 2.9B, which has 1.5 times the expert parameters and computation. In addition, DeepSeekMoE 2B nearly approaches the performance of its dense counterpart with the same number of total parameters, which set the upper bound of MoE models. Subsequently, we scale up DeepSeekMoE to 16B parameters and show that it achieves comparable performance with LLaMA2 7B, with only about 40% of computations. Further, our preliminary efforts to scale up DeepSeekMoE to 145B parameters consistently validate its substantial advantages over the GShard architecture, and show its performance comparable with DeepSeek 67B, using only 28.5% (maybe even 18.2%) of computations.

Computation and Language

What problem does this paper attempt to address?

This paper explores how to achieve a higher degree of specialization among experts in large-scale language models by improving the architecture of Mixture-of-Experts (MoE) through optimized computational efficiency. Existing MoE architectures, such as GShard, manage computational costs but suffer from knowledge mixing and redundancy, limiting expert specialization. To address this, the paper proposes the DeepSeekMoE architecture, which includes two main strategies:

1. Fine-grained expert segmentation: Expert segmentation into smaller parts and activation of a subset of them allows for more flexible expert combinations, enabling each expert to learn specific knowledge more accurately.

2. Shared expert isolation: Preserving a subset of experts as always-active shared experts for capturing and integrating common knowledge across different contexts, reducing knowledge redundancy among other experts.

Experiments demonstrate that DeepSeekMoE achieves comparable performance to the GShard 2.9B model with 1.5 times fewer expert parameters and computational resources on a 2B parameter scale. Furthermore, DeepSeekMoE 16B achieves performance similar to DeepSeek 7B with approximately 40% of the computational resources on a 16B parameter scale. The paper also shows that DeepSeekMoE consistently outperforms the GShard architecture on a 145B parameter scale, approaching the performance of DeepSeek 67B with only 28.5% (or even lower) computational resources.

In conclusion, DeepSeekMoE enhances the expert specialization of MoE models through innovative architectural design, achieving efficient and specialized parameter utilization while reducing computational costs while maintaining high performance.