Abstract:Multi-task learning (MTL) leverages a shared model to accomplish multiple tasks and facilitate knowledge transfer. Recent research on task arithmetic-based MTL demonstrates that merging the parameters of independently fine-tuned models can effectively achieve MTL. However, existing merging methods primarily seek a static optimal solution within the original model parameter space, which often results in performance degradation due to the inherent diversity among tasks and potential interferences. To address this challenge, in this paper, we propose a Weight-Ensembling Mixture of Experts (WEMoE) method for multi-task model merging. Specifically, we first identify critical (or sensitive) modules by analyzing parameter variations in core modules of Transformer-based models before and after finetuning. Then, our WEMoE statically merges non-critical modules while transforming critical modules into a mixture-of-experts (MoE) structure. During inference, expert modules in the MoE are dynamically merged based on input samples, enabling a more flexible and adaptive merging approach. Building on WEMoE, we further introduce an efficient-and-effective WEMoE (E-WEMoE) method, whose core mechanism involves eliminating non-essential elements in the critical modules of WEMoE and implementing shared routing across multiple MoE modules, thereby significantly reducing both the trainable parameters, the overall parameter count, and computational overhead of the merged model by WEMoE. Experimental results across various architectures and tasks demonstrate that both WEMoE and E-WEMoE outperform state-of-the-art (SOTA) model merging methods in terms of MTL performance, generalization, and robustness.

Parameter-efficient Weight Ensembling Facilitates Task-level Knowledge Transfer.

Towards a Unified View of Parameter-Efficient Transfer Learning

Enhanced Transfer Learning with Efficient Modeling and Adaptive Fusion of Knowledge Via Prompt Tuning

Merging Multi-Task Models via Weight-Ensembling Mixture of Experts

VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understanding

Parameter-Efficient Transfer Learning for NLP

Parameter-efficient Feature-Based Transfer for Paraphrase Identification.

HyperPELT: Unified Parameter-Efficient Language Model Tuning for Both Language and Vision-and-Language Tasks

Parameter-efficient Tuning of Large-scale Multimodal Foundation Model

Parameter-Efficient Fine-Tuning With Adapters

Revisit Parameter-Efficient Transfer Learning: A Two-Stage Paradigm

PEMT: Multi-Task Correlation Guided Mixture-of-Experts Enables Parameter-Efficient Transfer Learning

Optimal Parameter-Transfer Learning by Semiparametric Model Averaging

ScaLearn: Simple and Highly Parameter-Efficient Task Transfer by Learning to Scale

Weight Distillation: Transferring the Knowledge in Neural Network Parameters

Efficient Multi-Task and Transfer Reinforcement Learning with Parameter-Compositional Framework

Making Parameter-efficient Tuning More Efficient: A Unified Framework for Classification Tasks.

Efficient and Effective Weight-Ensembling Mixture of Experts for Multi-Task Model Merging

Boosting Inference Efficiency: Unleashing the Power of Parameter-Shared Pre-trained Language Models

$π$-Tuning: Transferring Multimodal Foundation Models with Optimal Multi-task Interpolation

PALT: Parameter-Lite Transfer of Language Models for Knowledge Graph Completion