Abstract:Federated multi-task learning (FMTL) has emerged as a promising framework for learning multiple tasks simultaneously with client-aware personalized models. While the majority of studies have focused on dealing with the non-independent and identically distributed (Non-IID) characteristics of client datasets, the issue of task heterogeneity has largely been overlooked. Dealing with task heterogeneity often requires complex models, making it impractical for federated learning in resource-constrained environments. In addition, the varying nature of these heterogeneous tasks introduces inductive biases, leading to interference during aggregation and potentially resulting in biased global models. To address these issues, we propose a hierarchical FMTL framework, referred to as FedBone, to facilitate the construction of large-scale models with improved generalization. FedBone leverages server-client split learning and gradient projection to split the entire model into two components: 1) a large-scale general model (referred to as the general model) on the cloud server, and 2) multiple task-specific models (referred to as client models) on edge clients, accommodating devices with limited compute power. To enhance the robustness of the large-scale general model, we incorporate the conflicting gradient projection technique into FedBone to rectify the skewed gradient direction caused by aggregating gradients from heterogeneous tasks. The proposed FedBone framework is evaluated on three benchmark datasets and one real ophthalmic dataset. The comprehensive experiments demonstrate that FedBone efficiently adapts to the heterogeneous local tasks of each client and outperforms existing federated learning algorithms in various dense prediction and classification tasks while utilizing off-the-shelf computational resources on the client side.

FedDGA: Federated Multi-Task Learning Based on Dynamic Guided Attention

FedDGP: Disentangling Global and Personal Models for Federated Learning

Federated Multitask Learning for HyperFace

HFedMTL: Hierarchical Federated Multi-Task Learning

A New Look and Convergence Rate of Federated Multi-Task Learning with Laplacian Regularization

Federated Multi-Task Learning under a Mixture of Distributions

Federated Multi-Task Learning on Non-IID Data Silos: An Experimental Study

FedBone: Towards Large-Scale Federated Multi-Task Learning

A New Look and Convergence Rate of Federated Multitask Learning With Laplacian Regularization

FedMPT: Federated Learning for Multiple Personalized Tasks Over Mobile Computing

Multi-Attribute Auction-Based Grouped Federated Learning

FedDA: Resource-adaptive Federated Learning with Dual-Alignment Aggregation Optimization for Heterogeneous Edge Devices

FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation

FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning

Dynamic Attention-based Communication-Efficient Federated Learning

FedHCA$^2$: Towards Hetero-Client Federated Multi-Task Learning

FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneity

FedU: A Unified Framework for Federated Multi-Task Learning with Laplacian Regularization

Federated Mutual Learning: a Collaborative Machine Learning Method for Heterogeneous Data, Models, and Objectives

MAS: Towards Resource-Efficient Federated Multiple-Task Learning

Fine-tuning Global Model Via Data-Free Knowledge Distillation for Non-IID Federated Learning