Abstract:Multi-task learning has been applied successfully in various applications. Recent research shows that the performance of multi-task learning methods could be improved by appropriately sharing model architectures. However, the existing work either identifies multi-task architecture manually based on prior knowledge, or simply uses an identical model structure for all tasks with a parameter sharing mechanism. In this paper, we propose a novel architecture search method to discover flexible and compact architectures for deep multi-task learning automatically, which not only extends the expressiveness of existing reinforcement learning-based neural architecture search methods, but also enhances the flexibility of existing hand-crafted multi-task learning methods. The discovered architecture shares structure and parameters adaptively to handle different levels of task relatedness, resulting in effectiveness improvement. In particular, for deep multi-task learning, we propose an architecture search space which includes a combination of partially shared modules at the low-level layer, and a set of task-specific modules with various depths at high-level layers. Secondly, a parameter generation mechanism is proposed to not only explore all possible cross-layer connections, but also reduce the search cost. Thirdly, we propose a task-specific shadow batch normalization mechanism to stabilize the training process and improve the search effectiveness. Finally, an auxiliary module is designed to guide the model training process. Experimental results demonstrate that the learned architectures outperform state-of-the-art methods with fewer learning parameters.

Learning Sparse Sharing Architectures for Multiple Tasks.

Task's Choice: Pruning-Based Feature Sharing (PBFS) for Multi-Task Learning.

MSSM: A Multiple-level Sparse Sharing Model for Efficient Multi-Task Learning

ESSR: Evolving Sparse Sharing Representation for Multi-task Learning

Adaptively Sharing Multi-Levels of Distributed Representations in Multi-Task Learning

Deep Multi-Task Learning with Shared Memory

Adaptive Hard Parameter Sharing Method Based on Multi-Task Deep Learning

Fitting and sharing multi-task learning

Multi-Task Learning Via Sharing Inexact Low-Rank Subspace

Learning Task Grouping and Overlap in Multi-task Learning

Sharing Knowledge in Multi-Task Deep Reinforcement Learning

Distributed Jointly Sparse Multitask Learning over Networks

Exploring Shared Structures and Hierarchies for Multiple NLP Tasks

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

Feature Partitioning for Efficient Multi-Task Architectures

MASR: A Model-Agnostic Sparse Routing Architecture for Arbitrary Order Feature Sharing in Multi-Task Learning.

Deep multi-task learning with flexible and compact architecture search

Adaptive Sharing for Image Classification.

Dual-mask: Progressively Sparse Multi-Task Architecture Learning

Multi-task Model and Feature Joint Learning

Efficient Computation Sharing for Multi-Task Visual Scene Understanding