Abstract:Distributed multi-task learning (MTL) is a learning paradigm where distributed users simultaneously learn multiple tasks by leveraging the correlations among tasks. However, distributed MTL suffers from a more severe communication bottleneck than single-task learning as more than one models need to be transmitted in the communication phase. To address this issue, we investigate the hierarchical MTL system where distributed users wish to jointly learn different learning models orchestrated by a central server with the help of multiple relays. We propose a coded distributed computing scheme for hierarchical MTL systems that jointly exploits the network topology and relays’ computing capability to create coded multicast opportunities to improve communication efficiency. We theoretically prove that the proposed scheme can significantly reduce the communication loads both in the uplink and downlink transmissions between relays and the server. To further illustrate the optimality of the proposed scheme, we derive information-theoretic lower bounds on the minimum uplink and downlink communication loads and prove that the gaps between achievable upper bounds and lower bounds are within the minimum number of connected users among all relays. In particular, when the network topology can be delicately designed, the proposed scheme can achieve the information-theoretic optimal communication loads. Experiments on real-world datasets show that our proposed scheme can greatly reduce the overall training time compared to the conventional hierarchical MTL scheme.

On Exploiting Network Topology for Hierarchical Coded Multi-task Learning

Coded Distributed Computing for Hierarchical Multi-task Learning

Communication-Efficient Coded Distributed Multi - Task Learning.

HFedMTL: Hierarchical Federated Multi-Task Learning

Distributed Learning of Predictive Structures from Multiple Tasks over Networks

Hierarchical Coded Matrix Multiplication in Heterogeneous Multihop Networks

Coded Parallelism for Distributed Deep Learning.

Distributed Networked Multi-task Learning

DEPHN: Different Expression Parallel Heterogeneous Network using virtual gradient optimization for Multi-task Learning

Network Clustering for Multi-task Learning

Topology Design of Network-Coding-Based Multicast Networks

Sequential Cooperative Distillation for Imbalanced Multi-Task Learning

An Adaptive Node Network Based on Multitask Deep Learning

Exploring Logically Dependent Multi task Learning with Causal Inference

Distributed Multi-Task Relationship Learning

Heterogeneous Coded Computation Across Heterogeneous Workers.

HirMTL: Hierarchical Multi-Task Learning for dense scene understanding

Two-Stage Coded Distributed Learning: A Dynamic Partial Gradient Coding Perspective

Communication-Efficient Topology Orchestration for Distributed Learning in UAV Networks

Grouped Multi-Task Learning with Hidden Tasks Enhancement

Distributed LT Codes on Multiple Layers Networks