Abstract:Solving multi-objective optimization problems for large deep neural networks is a challenging task due to the complexity of the loss landscape and the expensive computational cost of training and evaluating models. Efficient Pareto front approximation of large models enables multi-objective optimization for various tasks such as multi-task learning and trade-off analysis. Existing algorithms for learning Pareto set, including (1) evolutionary, hypernetworks, and hypervolume-maximization methods, are computationally expensive and have restricted scalability to large models; (2) Scalarization algorithms, where a separate model is trained for each objective ray, which is inefficient for learning the entire Pareto set and fails to capture the objective trade-offs effectively. Inspired by the recent success of model merging, we propose a practical and scalable approach to Pareto set learning problem via mixture of experts (MoE) based model fusion. By ensembling the weights of specialized single-task models, the MoE module can effectively capture the trade-offs between multiple objectives and closely approximate the entire Pareto set of large neural networks. Once the routers are learned and a preference vector is set, the MoE module can be unloaded, thus no additional computational cost is introduced during inference. We conduct extensive experiments on vision and language tasks using large-scale models such as CLIP-ViT and GPT-2. The experimental results demonstrate that our method efficiently approximates the entire Pareto front of large models. Using only hundreds of trainable parameters of the MoE routers, our method even has lower memory usage compared to linear scalarization and algorithms that learn a single Pareto optimal solution, and are scalable to both the number of objectives and the size of the model.

Mixture of Experts for Intelligent Networks: A Large Language Model-enabled Approach

Mixture of Experts for Network Optimization: A Large Language Model-enabled Approach

AT-MoE: Adaptive Task-planning Mixture of Experts via LoRA Approach

WDMoE: Wireless Distributed Large Language Models with Mixture of Experts

WDMoE: Wireless Distributed Mixture of Experts for Large Language Models

Routing Experts: Learning to Route Dynamic Experts in Multi-modal Large Language Models

HMoE: Heterogeneous Mixture of Experts for Language Modeling

LocMoE: A Low-Overhead MoE for Large Language Model Training

Theory of Mixture-of-Experts for Mobile Edge Computing

AdapMoE: Adaptive Sensitivity-based Expert Gating and Management for Efficient MoE Inference

Harder Tasks Need More Experts: Dynamic Routing in MoE Models

Intuition-aware Mixture-of-Rank-1-Experts for Parameter Efficient Finetuning

A Closer Look into Mixture-of-Experts in Large Language Models

Expert-Token Resonance: Redefining MoE Routing through Affinity-Driven Active Selection

Mixture of Experts in a Mixture of RL settings

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts

A Survey on Mixture of Experts

Generative AI Agents with Large Language Model for Satellite Networks via a Mixture of Experts Transmission

Large Language Model (LLM)-enabled In-context Learning for Wireless Network Optimization: A Case Study of Power Control

Towards Efficient Pareto Set Approximation via Mixture of Experts Based Model Fusion

Layerwise Recurrent Router for Mixture-of-Experts