Abstract:MTL is a learning paradigm that effectively leverages both task-specific and shared information to address multiple related tasks simultaneously. In contrast to STL, MTL offers a suite of benefits that enhance both the training process and the inference efficiency. MTL's key advantages encompass streamlined model architecture, performance enhancement, and cross-domain generalizability. Over the past twenty years, MTL has become widely recognized as a flexible and effective approach in various fields, including CV, NLP, recommendation systems, disease prognosis and diagnosis, and robotics. This survey provides a comprehensive overview of the evolution of MTL, encompassing the technical aspects of cutting-edge methods from traditional approaches to deep learning and the latest trend of pretrained foundation models. Our survey methodically categorizes MTL techniques into five key areas: regularization, relationship learning, feature propagation, optimization, and pre-training. This categorization not only chronologically outlines the development of MTL but also dives into various specialized strategies within each category. Furthermore, the survey reveals how the MTL evolves from handling a fixed set of tasks to embracing a more flexible approach free from task or modality constraints. It explores the concepts of task-promptable and -agnostic training, along with the capacity for ZSL, which unleashes the untapped potential of this historically coveted learning paradigm. Overall, we hope this survey provides the research community with a comprehensive overview of the advancements in MTL from its inception in 1997 to the present in 2023. We address present challenges and look ahead to future possibilities, shedding light on the opportunities and potential avenues for MTL research in a broad manner. This project is publicly available at

Identifying beneficial task relations for multi-task learning in deep neural networks

Multi-Task Learning in Natural Language Processing: An Overview

An Overview of Multi-Task Learning in Deep Neural Networks

Learning Functions to Study the Benefit of Multitask Learning

Multi-task learning for natural language processing in the 2020s: where are we going?

On Better Exploring and Exploiting Task Relationships in Multitask Learning: Joint Model and Feature Learning.

An End-to-End Scalable Iterative Sequence Tagging with Multi-Task Learning.

Modeling Output-Level Task Relatedness in Multi-Task Learning with Feedback Mechanism

Multiple Task Learning Using Iteratively Reweighted Least Square.

Traffic Flow and Speed Forecasting Through a Bayesian Deep Multi-Linear Relationship Network.

Multi-task Model and Feature Joint Learning

Distribution Matching for Multi-Task Learning of Classification Tasks: a Large-Scale Study on Faces & Beyond

Improving Gradient Trade-offs between Tasks in Multi-task Text Classification

Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Data

Task Grouping for Automated Multi-Task Machine Learning via Task Affinity Prediction

Robust Analysis of Multi-Task Learning Efficiency: New Benchmarks on Light-Weighed Backbones and Effective Measurement of Multi-Task Learning Challenges by Feature Disentanglement

Improving Multi-task Learning via Seeking Task-based Flat Regions

Unleashing the Power of Multi-Task Learning: A Comprehensive Survey Spanning Traditional, Deep, and Pretrained Foundation Model Eras

AdaTask: A Task-aware Adaptive Learning Rate Approach to Multi-task Learning