Abstract:Multi-task learning (MTL) has succeeded in various industrial applications by utilizing common knowledge among joint training tasks to enhance the generalization of MTL models, resulting in improved performance across all training tasks simultaneously. Unfortunately, training all tasks simultaneously often causes performance degradation compared to single-task models since different tasks might conflict with each other. Despite existing MTL methods that aim to mitigate task conflicts by manipulating task gradients at each iteration, they ignore the potential influence of noisy data from different batches on task gradients. Consequently, the current iteration's task gradient may not accurately reflect the task itself, leading to inadequate alleviation of the dilemma of task conflicts. Moreover, existing works seldom explore the potential source of task conflicts and merely pose an assumption. In this paper, we conduct an in-depth empirical investigation into the potential sources of performance degradation of MTL and find that task gradient conflict is one of the primary reasons for the performance degradation of tasks. Then, to address the task conflicts problem, we propose a novel gradient manipulation approach, namely MoCoGrad, which manipulates task gradients by leveraging the momentum information of the task to calibrate the gradients of conflicting tasks. In addition, we derive theoretical guarantees for the con-vergence of our proposed MoCoGrad and theoretically analyze the convergence rate of MoCoGrad. Finally, to evaluate the effectiveness of MoCoGrad, extensive experiments are conducted on six real-world datasets from different domains. Our approach yields the best performance across all tasks in all six MTL benchmarks, demonstrating the effectiveness and superiority of our method.

Improving Gradient Trade-offs between Tasks in Multi-task Text Classification

Improvable Gap Balancing for Multi-Task Learning

A Model-Agnostic Approach to Mitigate Gradient Interference for Multi-Task Learning

Multi-task Gradient Descent for Multi-Task Learning

GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning

Towards Task-Conflicts Momentum-Calibrated Approach for Multi-task Learning

Towards Impartial Multi-task Learning.

Dual-Balancing for Multi-Task Learning

Equitable Multi-task Learning

Improving Multi-task Learning via Seeking Task-based Flat Regions

Conflict-Averse Gradient Descent for Multi-task Learning

Fair Resource Allocation in Multi-Task Learning

Learning Boost by Exploiting the Auxiliary Task in Multi-task Domain

Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective

A Multi-task Learning Approach by Combining Derivative-Free and Gradient Methods.

AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning

MT-GBM: A Multi-Task Gradient Boosting Machine with Shared Decision Trees

A Multi-Task Gradient Descent Method for Multi-Label Learning

Controllable Pareto Multi-Task Learning

Learning to Resolve Conflicts in Multi-Task Learning

No More Tuning: Prioritized Multi-Task Learning with Lagrangian Differential Multiplier Methods