Abstract:In large language models like the Generative Pre-trained Transformer, the Mixture of Experts paradigm has emerged as a powerful technique for enhancing model expressiveness and accuracy. However, deploying GPT MoE models for parallel inference on distributed systems presents significant challenges, primarily due to the extensive Alltoall communication required for expert routing and aggregation. This communication bottleneck exacerbates the already complex computational landscape, hindering the efficient utilization of high-performance computing resources. In this paper, we propose a lightweight optimization technique called ExFlow, to largely accelerate the inference of these MoE models. We take a new perspective on alleviating the communication overhead by exploiting the inter-layer expert affinity. Unlike previous methods, our solution can be directly applied to pre-trained MoE models without any fine-tuning or accuracy degradation. By proposing a context-coherent expert parallelism on distributed systems, our design only uses one Alltoall communication to deliver the same functionality while previous methods all require two Alltoalls. By carefully examining the conditional probability in tokens' routing across multiple layers, we proved that pre-trained GPT MoE models implicitly exhibit a strong inter-layer expert affinity. We then design an efficient integer programming model to capture such features and show that by properly placing the experts on corresponding GPUs, we can reduce up to 67% cross-GPU routing latency. Our solution beats the cutting-edge MoE implementations with experts from 8 to 64, with up to 2.2x improvement in inference throughput. We further provide a detailed study of how the model implicitly acquires this expert affinity at the very early training stage and how this affinity evolves and stabilizes during training.

Using Mixture of Experts to Accelerate Dataset Distillation

Accelerating Dataset Distillation Via Model Augmentation

Towards Stable and Storage-efficient Dataset Distillation: Matching Convexified Trajectory

MoDE: A Mixture-of-Experts Model with Mutual Distillation among the Experts

Distributed Learning of Mixtures of Experts

Dataset Distillation by Matching Training Trajectories

Efficient Mixture of Experts Based on Large Language Models for Low-Resource Data Preprocessing

AST: Effective Dataset Distillation through Alignment with Smooth and High-Quality Expert Trajectories

Beyond Distillation: Task-level Mixture-of-Experts for Efficient Inference

D$^4$M: Dataset Distillation via Disentangled Diffusion Model

Latent Dataset Distillation with Diffusion Models

Efficient Dataset Distillation via Diffusion-Driven Patch Selection for Improved Generalization

Embarassingly Simple Dataset Distillation

Distilling Long-tailed Datasets

Dataset Distillation by Automatic Training Trajectories

Data Distillation Can Be Like Vodka: Distilling More Times For Better Quality

Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation

MoEC: Mixture of Expert Clusters

Variational Distillation of Diffusion Policies into Mixture of Experts

Task-Specific Expert Pruning for Sparse Mixture-of-Experts