Abstract:Constructing useful representations across a large number of tasks is a key requirement for sample-efficient intelligent systems. A traditional idea in multitask learning (MTL) is building a shared representation across tasks which can then be adapted to new tasks by tuning last layers. A desirable refinement of using a shared one-fits-all representation is to construct task-specific representations. To this end, recent PathNet/muNet architectures represent individual tasks as pathways within a larger supernet. The subnetworks induced by pathways can be viewed as task-specific representations that are composition of modules within supernet's computation graph. This work explores the pathways proposal from the lens of statistical learning: We first develop novel generalization bounds for empirical risk minimization problems learning multiple tasks over multiple paths (Multipath MTL). In conjunction, we formalize the benefits of resulting multipath representation when adapting to new downstream tasks. Our bounds are expressed in terms of Gaussian complexity, lead to tangible guarantees for the class of linear representations, and provide novel insights into the quality and benefits of a multipath representation. When computation graph is a tree, Multipath MTL hierarchically clusters the tasks and builds cluster-specific representations. We provide further discussion and experiments for hierarchical MTL and rigorously identify the conditions under which Multipath MTL is provably superior to traditional MTL approaches with shallow supernets.

Tree-Like Branching Network for Multi-class Classification

CMNN: Coupled Modular Neural Network.

A Bayesian Network nearest k-labels method for Multi-label classification

Deep Multi-task Learning for Hierarchical Classification

Cumulative Dual-Branch Network Framework for Long-Tailed Multi-Class Classification

Connectivity Learning in Multi-Branch Networks

Traffic Flow and Speed Forecasting Through a Bayesian Deep Multi-Linear Relationship Network.

A Closer Look at Branch Classifiers of Multi-exit Architectures

Learning What to Share: Leaky Multi-Task Network for Text Classification.

Multi-Task Learning Network for Landmark Detection in Anatomical Tree Structures.

Network Transplanting (extended abstract)

A Tri-Branch Network with Prototype-aware Matching for Universal Category Discovery

Deep collaborative multi-task network: A human decision process inspired model for hierarchical image classification

Learning to Branch with Tree-aware Branching Transformers

Deep Multi-task Learning for Large-Scale Image Classification

Hierarchical Learning of Tree Classifiers for Large-Scale Plant Species Identification

Multi-branch Collaborative Learning Network for 3D Visual Grounding

Multitask Learning With Enhanced Modules.

Provable Pathways: Learning Multiple Tasks over Multiple Paths

Association Graph Learning for Multi-Task Classification with Category Shifts

Hierarchical Inter-Attention Network for Document Classification with Multi-Task Learning.