Abstract:The availability of performant pre-trained models has led to a proliferation of fine-tuned expert models that are specialized to particular domains. This has enabled the creation of powerful and adaptive routing-based "Model MoErging" methods with the goal of using expert modules to create an aggregate system with improved performance or generalization. However, existing MoErging methods often prioritize generalization to unseen tasks at the expense of performance on held-in tasks, which limits its practical applicability in real-world deployment scenarios. We observe that current token-level routing mechanisms neglect the global semantic context of the input task. This token-wise independence hinders effective expert selection for held-in tasks, as routing decisions fail to incorporate the semantic properties of the task. To address this, we propose, Global and Local Instruction Driven Expert Router (GLIDER) that integrates a multi-scale routing mechanism, encompassing a semantic global router and a learned local router. The global router leverages LLM's advanced reasoning capabilities for semantic-related contexts to enhance expert selection. Given the input query and LLM, the router generates semantic task instructions that guide the retrieval of the most relevant experts across all layers. This global guidance is complemented by a local router that facilitates token-level routing decisions within each module, enabling finer control and enhanced performance on unseen tasks. Our experiments using T5-based models for T0 and FLAN tasks demonstrate that GLIDER achieves substantially improved held-in performance while maintaining strong generalization on held-out tasks. We also perform ablations experiments to dive deeper into the components of GLIDER. Our experiments highlight the importance of our multi-scale routing that leverages LLM-driven semantic reasoning for MoErging methods.

Learning to Route Among Specialized Experts for Zero-Shot Generalization

Glider: Global and Local Instruction-Driven Expert Router

HyperRouter: Towards Efficient Training and Inference of Sparse Mixture of Experts

Efficient Routing in Sparse Mixture-of-Experts

Multi-Head Adapter Routing for Cross-Task Generalization

HyperLoRA: Efficient Cross-task Generalization Via Constrained Low-Rank Adapters Generation

Soft Merging of Experts with Adaptive Routing

Turn Waste into Worth: Rectifying Top-$k$ Router of MoE

Ada-K Routing: Boosting the Efficiency of MoE-based LLMs

HyperMoE: Towards Better Mixture of Experts via Transferring Among Experts

Exploiting Inter-Layer Expert Affinity for Accelerating Mixture-of-Experts Model Inference

Expert-Token Resonance: Redefining MoE Routing through Affinity-Driven Active Selection

Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Models

Model Selection for Generalized Zero-shot Learning

On the effectiveness of discrete representations in sparse mixture of experts

Multi-Task Learning for Routing Problem with Cross-Problem Zero-Shot Generalization

Prototype-Augmented Self-Supervised Generative Network for Generalized Zero-Shot Learning.

More Experts Than Galaxies: Conditionally-overlapping Experts With Biologically-Inspired Fixed Routing

An Expert is Worth One Token: Synergizing Multiple Expert LLMs as Generalist via Expert Token Routing

GOAL: A Generalist Combinatorial Optimization Agent Learner

Layerwise Recurrent Router for Mixture-of-Experts