Abstract:In large language models like the Generative Pre-trained Transformer, the Mixture of Experts paradigm has emerged as a powerful technique for enhancing model expressiveness and accuracy. However, deploying GPT MoE models for parallel inference on distributed systems presents significant challenges, primarily due to the extensive Alltoall communication required for expert routing and aggregation. This communication bottleneck exacerbates the already complex computational landscape, hindering the efficient utilization of high-performance computing resources. In this paper, we propose a lightweight optimization technique called ExFlow, to largely accelerate the inference of these MoE models. We take a new perspective on alleviating the communication overhead by exploiting the inter-layer expert affinity. Unlike previous methods, our solution can be directly applied to pre-trained MoE models without any fine-tuning or accuracy degradation. By proposing a context-coherent expert parallelism on distributed systems, our design only uses one Alltoall communication to deliver the same functionality while previous methods all require two Alltoalls. By carefully examining the conditional probability in tokens' routing across multiple layers, we proved that pre-trained GPT MoE models implicitly exhibit a strong inter-layer expert affinity. We then design an efficient integer programming model to capture such features and show that by properly placing the experts on corresponding GPUs, we can reduce up to 67% cross-GPU routing latency. Our solution beats the cutting-edge MoE implementations with experts from 8 to 64, with up to 2.2x improvement in inference throughput. We further provide a detailed study of how the model implicitly acquires this expert affinity at the very early training stage and how this affinity evolves and stabilizes during training.

APTMoE: Affinity-Aware Pipeline Tuning for MoE Models on Bandwidth-Constrained GPU Nodes

MPMoE: Memory Efficient MoE for Pre-Trained Models With Adaptive Pipeline Parallelism

Towards MoE Deployment: Mitigating Inefficiencies in Mixture-of-Expert (moe) Inference

MPipeMoE: Memory Efficient MoE for Pre-trained Models with Adaptive Pipeline Parallelism

Pipeline MoE: A Flexible MoE Implementation with Pipeline Parallelism

PipeMoE: Accelerating Mixture-of-Experts through Adaptive Pipelining.

HEXA-MoE: Efficient and Heterogeneous-aware MoE Acceleration with ZERO Computation Redundancy

MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUs

ScheMoE

ScheMoE: An Extensible Mixture-of-Experts Distributed Training System with Tasks Scheduling.

EPS-MoE: Expert Pipeline Scheduler for Cost-Efficient MoE Inference

AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference

Pre-gated MoE: An Algorithm-System Co-Design for Fast and Scalable Mixture-of-Expert Inference

Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts

Optimizing Mixture-of-Experts Inference Time Combining Model Deployment and Communication Scheduling

FASTERMOE: Modeling and Optimizing Training of Large-Scale Dynamic Pre-Trained Models

HetuMoE: An Efficient Trillion-scale Mixture-of-Expert Distributed Training System

MoE-Infinity: Offloading-Efficient MoE Model Serving

Pro-Prophet: A Systematic Load Balancing Method for Efficient Parallel Training of Large-scale MoE Models

ProMoE: Fast MoE-based LLM Serving using Proactive Caching