Abstract:In recent years, with the rapid application of large language models across various fields, the scale of these models has gradually increased, and the resources required for their pre-training have grown exponentially. Training an LLM from scratch will cost a lot of computation resources while scaling up from a smaller model is a more efficient approach and has thus attracted significant attention. In this paper, we present AquilaMoE, a cutting-edge bilingual 8*16B Mixture of Experts (MoE) language model that has 8 experts with 16 billion parameters each and is developed using an innovative training methodology called EfficientScale. This approach optimizes performance while minimizing data requirements through a two-stage process. The first stage, termed Scale-Up, initializes the larger model with weights from a pre-trained smaller model, enabling substantial knowledge transfer and continuous pretraining with significantly less data. The second stage, Scale-Out, uses a pre-trained dense model to initialize the MoE experts, further enhancing knowledge transfer and performance. Extensive validation experiments on 1.8B and 7B models compared various initialization schemes, achieving models that maintain and reduce loss during continuous pretraining. Utilizing the optimal scheme, we successfully trained a 16B model and subsequently the 8*16B AquilaMoE model, demonstrating significant improvements in performance and training efficiency.

What problem does this paper attempt to address?

The paper primarily addresses several key issues in large-scale language model training: 1. **Resource Consumption**: With the widespread application of large language models in various fields, the model size is gradually increasing, and the resources required for pre-training are growing exponentially. Training a large language model from scratch requires a significant amount of computational resources, whereas expanding from a smaller model is more efficient. 2. **Data Requirements and Computational Costs**: Training large-scale models (including Mixture of Experts (MoE) architectures) faces resource-intensive challenges in data collection and processing, as well as high time and computational costs. Traditional training methods require a large amount of data, which is not only time-consuming but also demands high hardware specifications, making it difficult for resource-limited institutions to achieve. 3. **Training Efficiency and Performance Optimization**: Training large-scale models from scratch can take weeks or even months, delaying the experimentation and iteration process. Additionally, improper initialization or inefficient training strategies can lead to poor model performance, resulting in resource wastage. To address the above issues, the paper proposes AquilaMoE, a bilingual 8*16B Mixture of Experts (MoE) language model constructed using an innovative training method called EfficientScale. EfficientScale optimizes model performance and minimizes data requirements through a two-stage process: - **Scale-Up Stage**: Utilizes the weights of a smaller pre-trained model to initialize a larger model, thereby achieving knowledge transfer and performing continuous pre-training with significantly less data. - **Scale-Out Stage**: Uses the pre-trained dense model to initialize MoE experts, further enhancing knowledge transfer and performance. Through these methods, the researchers successfully trained a model with 16B parameters, followed by training the 8*16B parameter AquilaMoE model, achieving significant improvements in both model performance and training efficiency.

AquilaMoE: Efficient Training for MoE Models with Scale-Up and Scale-Out Strategies

Llama 3 Meets MoE: Efficient Upcycling

Toward Inference-optimal Mixture-of-Expert Large Language Models

MC-MoE: Mixture Compressor for Mixture-of-Experts LLMs Gains More

Scaling Laws Across Model Architectures: A Comparative Analysis of Dense and MoE Models in Large Language Models

LocMoE: A Low-Overhead MoE for Large Language Model Training

Dense Training, Sparse Inference: Rethinking Training of Mixture-of-Experts Language Models

LLaMA-MoE: Building Mixture-of-Experts from LLaMA with Continual Pre-training

U2++ MoE: Scaling 4.7x parameters with minimal impact on RTF

LSH-MoE: Communication-efficient MoE Training via Locality-Sensitive Hashing

Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts

JetMoE: Reaching Llama2 Performance with 0.1M Dollars

OLMoE: Open Mixture-of-Experts Language Models

Revisiting MoE and Dense Speed-Accuracy Comparisons for LLM Training

SwapMoE: Serving Off-the-shelf MoE-based Large Language Models with Tunable Memory Budget

Mixture of Diverse Size Experts

Upcycling Large Language Models into Mixture of Experts

OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models

MoE-LPR: Multilingual Extension of Large Language Models through Mixture-of-Experts with Language Priors Routing

Not All Experts are Equal: Efficient Expert Pruning and Skipping for Mixture-of-Experts Large Language Models

Memory Augmented Language Models through Mixture of Word Experts