Abstract:Model merging has gained increasing attention as an efficient and effective technique for integrating task-specific weights from various tasks into a unified multi-task model without retraining or additional data. As a representative approach, Task Arithmetic (TA) has demonstrated that combining task vectors through arithmetic operations facilitates efficient capability transfer between different tasks. In this framework, task vectors are obtained by subtracting the parameter values of a pre-trained model from those of individually fine-tuned models initialized from it. Despite the notable effectiveness of TA, interference among task vectors can adversely affect the performance of the merged model. In this paper, we relax the constraints of Task Arithmetic Property and propose Task Consistency Property, which can be regarded as being free from task interference. Through theoretical derivation, we show that such a property can be approximately achieved by seeking orthogonal task vectors. Guiding by this insight, we propose Adaptive Weight Disentanglement (AWD), which decomposes traditional task vectors into a redundant vector and several disentangled task vectors. The primary optimization objective of AWD is to achieve orthogonality among the disentangled task vectors, thereby closely approximating the desired solution. Notably, these disentangled task vectors can be seamlessly integrated into existing merging methodologies. Experimental results demonstrate that our AWD consistently and significantly improves upon previous merging approaches, achieving state-of-the-art results. Our code is available at \href{<a class="link-external link-https" href="https://github.com/FarisXiong/AWD.git" rel="external noopener nofollow">this https URL</a>}{<a class="link-external link-https" href="https://github.com/FarisXiong/AWD.git" rel="external noopener nofollow">this https URL</a>}.

ForkMerge: Mitigating Negative Transfer in Auxiliary-Task Learning

Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain

FairBranch: Mitigating Bias Transfer in Fair Multi-task Learning

Loss-Balanced Task Weighting to Reduce Negative Transfer in Multi-Task Learning

A Model-Agnostic Approach to Mitigate Gradient Interference for Multi-Task Learning

Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries

Multi-Task Distillation: Towards Mitigating the Negative Transfer in Multi-Task Learning

Unprejudiced Training Auxiliary Tasks Makes Primary Better: A Multitask Learning Perspective

Generalized Block-Diagonal Structure Pursuit: Learning Soft Latent Task Assignment Against Negative Transfer.

Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective

AdaMerging: Adaptive Model Merging for Multi-Task Learning

Gradient Coordination for Quantifying and Maximizing Knowledge Transference in Multi-Task Learning

Mitigate Negative Transfer with Similarity Heuristic Lifelong Prompt Tuning

AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning

Adaptive Hard Parameter Sharing Method Based on Multi-Task Deep Learning

ATM: Improving Model Merging by Alternating Tuning and Merging

GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning

Task Uncertainty Loss Reduce Negative Transfer in Asymmetric Multi-task Feature Learning

Multi-Task Model Merging via Adaptive Weight Disentanglement

Joint Data-Task Generation for Auxiliary Learning.

An Adaptive Node Network Based on Multitask Deep Learning