Abstract:In multi-task learning, a major challenge springs from a notorious issue known as negative transfer, which refers to the phenomenon that sharing the knowledge with dissimilar and hard tasks often results in a worsened performance. To circumvent this issue, we propose a novel multi-task learning method, which simultaneously learns latent task representations and a block-diagonal Latent Task Assignment Matrix (LTAM). Different from most of the previous work, pursuing the Block-Diagonal structure of LTAM (assigning latent tasks to output tasks) alleviates negative transfer via collaboratively grouping latent tasks and output tasks such that inter-group knowledge transfer and sharing is suppressed. This goal is challenging, since 1) our notion of Block-Diagonal Property extends the traditional notion for square matrices where the $i$-th column and the $i$-th column represents the same concept; 2) marginal constraints on rows and columns are also required for avoiding isolated latent/output tasks. Facing such challenges, we propose a novel regularizer by means of an equivalent spectral condition realizing this generalized block-diagonal property. Practically, we provide a relaxation scheme which improves the flexibility of the model. With the objective function given, we then propose an alternating optimization method, which not only tells how negative transfer is alleviated in our method but also reveals an interesting connection between our method and the optimal transport problem. Finally, the method is demonstrated on a simulation dataset, three real-world benchmark datasets and further applied to personalized attribute predictions.

Phase Transitions in Transfer Learning for High-Dimensional Perceptrons

Features are fate: a theory of transfer learning in high-dimensional regression

Characterizing and Avoiding Negative Transfer

The Common Intuition to Transfer Learning Can Win or Lose: Case Studies for Linear Regression

Rethinking Two Consensuses of the Transferability in Deep Learning

A Visual Analytical Approach for Transfer Learning in Classification.

A Decade Survey of Transfer Learning (2010–2020)

Frozen Overparameterization: A Double Descent Perspective on Transfer Learning of Deep Neural Networks

A Survey on Negative Transfer

Statistical mechanics of transfer learning in fully-connected networks in the proportional limit

Low-dimensional Intrinsic Dimension Reveals a Phase Transition in Gradient-Based Learning of Deep Neural Networks

Negative Transfer Detection in Transductive Transfer Learning

Towards A Unified Understanding and Improving of Adversarial Transferability

Transfer learning of phase transitions in percolation and directed percolation

A Unified Approach to Interpreting and Boosting Adversarial Transferability

Generalization ability of a perceptron with non-monotonic transfer function

Tractability from overparametrization: the example of the negative perceptron

Online Transfer Learning: Negative Transfer and Effect of Prior Knowledge

Generalized Block-Diagonal Structure Pursuit: Learning Soft Latent Task Assignment Against Negative Transfer.

A Novel Mapping Between Machine Learning and Phase Transition

TransBO: Hyperparameter Optimization Via Two-Phase Transfer Learning