Abstract:Multi-Task Learning (MTL) networks have emerged as a promising method for transferring learned knowledge across different tasks. However, MTL must deal with challenges such as: overfitting to low resource tasks, catastrophic forgetting, and negative task transfer, or learning interference. Often, in Natural Language Processing (NLP), a separate model per task is needed to obtain the best performance. However, many fine-tuning approaches are both parameter inefficient, i.e., potentially involving one new model per task, and highly susceptible to losing knowledge acquired during pretraining. We propose a novel Transformer architecture consisting of a new conditional attention mechanism as well as a set of task-conditioned modules that facilitate weight sharing. Through this construction (a hypernetwork adapter), we achieve more efficient parameter sharing and mitigate forgetting by keeping half of the weights of a pretrained model fixed. We also use a new multi-task data sampling strategy to mitigate the negative effects of data imbalance across tasks. Using this approach, we are able to surpass single task fine-tuning methods while being parameter and data efficient (using around 66% of the data for weight updates). Compared to other BERT Large methods on GLUE, our 8-task model surpasses other Adapter methods by 2.8% and our 24-task model outperforms by 0.7-1.0% models that use MTL and single task fine-tuning. We show that a larger variant of our single multi-task model approach performs competitively across 26 NLP tasks and yields state-of-the-art results on a number of test and development sets. Our code is publicly available at https://github.com/CAMTL/CA-MTL.

Transductive Auxiliary Task Self-Training for Neural Multi-Task Models

Hierarchical and Bidirectional Joint Multi-Task Classifiers for Natural Language Understanding

Iterative Task-adaptive Pretraining for Unsupervised Word Alignment

Improving Speech Translation by Understanding and Learning from the Auxiliary Text Translation Task

Transfer and Multi-Task Learning for Noun-Noun Compound Interpretation

Free Lunches in Auxiliary Learning: Exploiting Auxiliary Labels with Negligibly Extra Inference Cost

Improving End-to-End Text Image Translation From the Auxiliary Text Translation Task

Constructing Multiple Tasks for Augmentation: Improving Neural Image Classification with K-means Features

Aux-NAS: Exploiting Auxiliary Labels with Negligibly Extra Inference Cost

Joint Data-Task Generation for Auxiliary Learning.

Enabling Asymmetric Knowledge Transfer in Multi-Task Learning with Self-Auxiliaries

Identifying beneficial task relations for multi-task learning in deep neural networks

Multi-Method Self-Training: Improving Code Generation With Text, And Vice Versa

Multidimensional Transformation-Based Learning

GradTS: A Gradient-Based Automatic Auxiliary Task Selection Method Based on Transformer Networks

Disentangled Latent Spaces Facilitate Data-Driven Auxiliary Learning

Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data

Auxiliary Learning with Joint Task and Data Scheduling.

A Self-Attentional Neural Architecture for Code Completion with Multi-Task Learning.

An End-to-End Scalable Iterative Sequence Tagging with Multi-Task Learning.

Learning from Semantically Dependent Multi-Tasks