Abstract:The size of deep learning models has been increasing to enhance model quality. The linear increase in training computation budget with model size means that training an extremely large-scale model is exceedingly time-consuming. Recently, the Mixture of Expert (MoE) has drawn significant attention as it can scale models to extra-large sizes with a stable computation budget. However, inefficient distributed training of large-scale MoE models hinders their broader application. Specifically, a considerable dynamic load imbalance occurs among devices during training, significantly reducing throughput. Several load-balancing works have been proposed to address the challenge. System-level solutions draw more attention for their hardware affinity and non-disruption of model convergence compared to algorithm-level ones. However, they are troubled by high communication costs and poor communication-computation overlapping. To address these challenges, we propose a systematic load-balancing method, Pro-Prophet, which consists of a planner and a scheduler for efficient parallel training of large-scale MoE models. To adapt to the dynamic load imbalance, we profile training statistics and use them to design Pro-Prophet. For lower communication volume, Pro-Prophet planner determines a series of lightweight load-balancing strategies and efficiently searches for a communication-efficient one for training based on the statistics. For sufficient overlapping of communication and computation, Pro-Prophet scheduler schedules the data-dependent operations based on the statistics and operation features, further improving the training throughput. Experimental results indicate that Pro-Prophet achieves up to 2.66x speedup compared to Deepspeed-MoE and FasterMoE. Additionally, Pro-Prophet achieves a load-balancing enhancement of up to 11.01x when compared to FasterMoE.

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

MPMoE: Memory Efficient MoE for Pre-Trained Models With Adaptive Pipeline Parallelism

PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining.

Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism

Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

FASTERMOE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference

FasterMoE

ScheMoE

Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (moe) Inference

SwapMoE: Efficient Memory-Constrained Serving of Large Sparse MoE Models Via Dynamic Expert Pruning and Swapping

Condense, Don't Just Prune: Enhancing Efficiency and Performance in MoE Layer Pruning

DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scale

Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models

HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System

FlexMoE: Scaling Large-scale Sparse Pre-trained Model Training via Dynamic Device Placement

FastMoE: A Fast Mixture-of-Expert Training System

ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling.

ProMoE: Fast MoE-based LLM Serving using Proactive Caching

Communication-Efficient Sparsely-Activated Model Training via Sequence Migration and Token Condensation