Abstract:With the development of edge networks and mobile computing, the need to serve heterogeneous data sources at the network edge requires the design of new distributed machine learning mechanisms. As a prevalent approach, Federated Learning (FL) employs parameter-sharing and gradient-averaging between clients and a server. Despite its many favorable qualities, such as convergence and data-privacy guarantees, it is well-known that classic FL fails to address the challenge of data heterogeneity and computation heterogeneity across clients. Most existing works that aim to accommodate such sources of heterogeneity stay within the FL operation paradigm, with modifications to overcome the negative effect of heterogeneous data. In this work, as an alternative paradigm, we propose a Multi-Task Split Learning (MTSL) framework, which combines the advantages of Split Learning (SL) with the flexibility of distributed network architectures. In contrast to the FL counterpart, in this paradigm, heterogeneity is not an obstacle to overcome, but a useful property to take advantage of. As such, this work aims to introduce a new architecture and methodology to perform multi-task learning for heterogeneous data sources efficiently, with the hope of encouraging the community to further explore the potential advantages we reveal. To support this promise, we first show through theoretical analysis that MTSL can achieve fast convergence by tuning the learning rate of the server and clients. Then, we compare the performance of MTSL with existing multi-task FL methods numerically on several image classification datasets to show that MTSL has advantages over FL in training speed, communication cost, and robustness to heterogeneous data.

Exploiting Feature Heterogeneity for Improved Generalization in Federated Multi-task Learning

Exploiting High-Order Information in Heterogeneous Multi-Task Feature Learning.

Federated Multi-Task Learning under a Mixture of Distributions

FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning

Learning to Generalize in Heterogeneous Federated Networks

A New Look and Convergence Rate of Federated Multi-Task Learning with Laplacian Regularization

Non-Federated Multi-Task Split Learning for Heterogeneous Sources

A New Look and Convergence Rate of Federated Multitask Learning With Laplacian Regularization

Towards Efficient Model-Heterogeneity Federated Learning for Large Models

Towards Addressing Heterogeneity Of Data In Federated Learning

FedDGA: Federated Multi-Task Learning Based on Dynamic Guided Attention

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

Federated Mutual Learning: a Collaborative Machine Learning Method for Heterogeneous Data, Models, and Objectives

FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning

Federated Model Heterogeneous Matryoshka Representation Learning

Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training

Federated Multi-Task Learning with Non-Stationary and Heterogeneous Data in Wireless Networks

Privacy preserving federated learning for full heterogeneity

FedTSA: A Cluster-based Two-Stage Aggregation Method for Model-heterogeneous Federated Learning

FedPer++: Toward Improved Personalized Federated Learning on Heterogeneous and Imbalanced Data

Completely Heterogeneous Federated Learning