Abstract:Multi-task learning (MTL) has succeeded in various industrial applications by utilizing common knowledge among joint training tasks to enhance the generalization of MTL models, resulting in improved performance across all training tasks simultaneously. Unfortunately, training all tasks simultaneously often causes performance degradation compared to single-task models since different tasks might conflict with each other. Despite existing MTL methods that aim to mitigate task conflicts by manipulating task gradients at each iteration, they ignore the potential influence of noisy data from different batches on task gradients. Consequently, the current iteration's task gradient may not accurately reflect the task itself, leading to inadequate alleviation of the dilemma of task conflicts. Moreover, existing works seldom explore the potential source of task conflicts and merely pose an assumption. In this paper, we conduct an in-depth empirical investigation into the potential sources of performance degradation of MTL and find that task gradient conflict is one of the primary reasons for the performance degradation of tasks. Then, to address the task conflicts problem, we propose a novel gradient manipulation approach, namely MoCoGrad, which manipulates task gradients by leveraging the momentum information of the task to calibrate the gradients of conflicting tasks. In addition, we derive theoretical guarantees for the con-vergence of our proposed MoCoGrad and theoretically analyze the convergence rate of MoCoGrad. Finally, to evaluate the effectiveness of MoCoGrad, extensive experiments are conducted on six real-world datasets from different domains. Our approach yields the best performance across all tasks in all six MTL benchmarks, demonstrating the effectiveness and superiority of our method.

A Multi-task Learning Approach by Combining Derivative-Free and Gradient Methods.

Multi-Task Learning with Multi-Task Optimization

Multi-Task Learning as Multi-Objective Optimization

Conflict-Averse Gradient Descent for Multi-task Learning

GDOD: Effective Gradient Descent using Orthogonal Decomposition for Multi-Task Learning

Learning Incoherent Sparse and Low-Rank Patterns from Multiple Tasks

Variable Selection and Task Grouping for Multi-Task Learning

Gradient Coordination for Quantifying and Maximizing Knowledge Transference in Multi-Task Learning

A Multi-Task Gradient Descent Method for Multi-Label Learning

Proactive Gradient Conflict Mitigation in Multi-Task Learning: A Sparse Training Perspective

Gradient Surgery for Multi-Task Learning

Do Current Multi-Task Optimization Methods in Deep Learning Even Help?

A Model-Agnostic Approach to Mitigate Gradient Interference for Multi-Task Learning

Low-Rank Deep Convolutional Neural Network for Multi-Task Learning

Pareto Multi-Task Learning

Efficiently Identifying Task Groupings for Multi-Task Learning

Improving Gradient Trade-offs between Tasks in Multi-task Text Classification

Adaptive multi-gradient methods for quasiconvex vector optimization and applications to multi-task learning

Multi-task Learning with Gradient Communication

Multi-task Model and Feature Joint Learning

Towards Task-Conflicts Momentum-Calibrated Approach for Multi-task Learning