Abstract:Distributed multi-task learning (MTL) is a learning paradigm where distributed users simultaneously learn multiple tasks by leveraging the correlations among tasks. However, distributed MTL suffers from a more severe communication bottleneck than single-task learning as more than one models need to be transmitted in the communication phase. To address this issue, we investigate the hierarchical MTL system where distributed users wish to jointly learn different learning models orchestrated by a central server with the help of multiple relays. We propose a coded distributed computing scheme for hierarchical MTL systems that jointly exploits the network topology and relays’ computing capability to create coded multicast opportunities to improve communication efficiency. We theoretically prove that the proposed scheme can significantly reduce the communication loads both in the uplink and downlink transmissions between relays and the server. To further illustrate the optimality of the proposed scheme, we derive information-theoretic lower bounds on the minimum uplink and downlink communication loads and prove that the gaps between achievable upper bounds and lower bounds are within the minimum number of connected users among all relays. In particular, when the network topology can be delicately designed, the proposed scheme can achieve the information-theoretic optimal communication loads. Experiments on real-world datasets show that our proposed scheme can greatly reduce the overall training time compared to the conventional hierarchical MTL scheme.

Scale-Aware Task Message Transferring for Multi-Task Learning

Adaptive Task-Wise Message Passing for Multi-Task Learning: A Spatial Interaction Perspective

MTI-Net: Multi-Scale Task Interaction Networks for Multi-Task Learning

Learning Multi-Task Communication with Message Passing for Sequence Learning.

HirMTL: Hierarchical Multi-Task Learning for dense scene understanding

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

Multi-scale Matching Networks for Semantic Correspondence

Traffic Flow and Speed Forecasting Through a Bayesian Deep Multi-Linear Relationship Network.

Hierarchical Task-aware Multi-Head Attention Network

Exploring the Role of Task Transferability in Large-Scale Multi-Task Learning

Task-Interaction-Free Multi-Task Learning with Efficient Hierarchical Feature Representation

MSSM: A Multiple-level Sparse Sharing Model for Efficient Multi-Task Learning

Multi-Task Label Discovery via Hierarchical Task Tokens for Partially Annotated Dense Predictions

Adaptively Sharing Multi-Levels of Distributed Representations in Multi-Task Learning

Multi-task Learning over Graph Structures.

Multi-task Learning with Deformable Convolution

MTMamba++: Enhancing Multi-Task Dense Scene Understanding via Mamba-Based Decoders

Network Clustering for Multi-task Learning

Robust Estimator Based Adaptive Multi-Task Learning

Multi-Task Structural Learning using Local Task Similarity induced Neuron Creation and Removal

On Exploiting Network Topology for Hierarchical Coded Multi-task Learning