Abstract:Federated Learning (FL) is an emerging training framework for machine learning to protect data privacy without accessing the original data from each client. However, the participating clients have different computing resources in FL. Clients with insufficient resources may not cooperate in the training due to hardware limitations. The restricted computing speeds may also slow down the overall computing time. In addition, the Non-IID problem happens when data distributions of the clients are varied, which results in lower performance for training. To overcome these problems, this paper proposes a FedCOLA approach to adapt various data distributions among heterogeneous clients. By introducing the feature concatenation and local update mechanism, FedCOLA supports different clients to train the model with different layers. Both communication load and time delay during collaborative training can be reduced. Combined with the adaptive auxiliary model and the personalized model, FedCOLA further improves the testing accuracy under various Non-IID data distributions. To evaluate the performance, this paper considers the effects and analysis of different Non-IID data distributions on distinct methods. The empirical results show that FedCOLA improves the accuracy by 5%, reduces 57% rounds to achieve the same accuracy, and reduces the communication load by 77% in the extremely imbalanced data distribution. Compared with the state-of-the-art methods in a real deployment of heterogeneous clients, FedCOLA reduces the time consumption by 70% to achieve the same accuracy and by 30% to complete 200 training rounds. In conclusion, the proposed FedCOLA not only accommodates various Non-IID data distributions but also supports the heterogeneous clients to train the model of different layers with a significant reduction of the time delay and communication load.

CoFED: Cross-silo Heterogeneous Federated Multi-task Learning via Co-training

Cross-silo Heterogeneous Model Federated Multitask Learning

FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation

Heterogeneous Federated Learning with Cross-layer Model Fusion.

FedCod: An Efficient Communication Protocol for Cross-Silo Federated Learning with Coding

Federated mutual learning

FedCML: Federated Clustering Mutual Learning with non-IID Data.

Completely Heterogeneous Federated Learning

Cross-Training with Multi-View Knowledge Fusion for Heterogenous Federated Learning

Cooperative Multi-Model Training for Personalized Federated Learning over Heterogeneous Devices

Fair Concurrent Training of Multiple Models in Federated Learning

FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement

FedCos: A Scene-Adaptive Enhancement for Federated Learning

FedCOLA: Federated Learning with Heterogeneous Feature Concatenation and Local Acceleration for Non-IID Data

CE-Fed: Communication Efficient Multi-Party Computation Enabled Federated Learning.

FedCo: Self-Supervised Learning in Federated Learning with Momentum Contrast.

Federated Mutual Learning: a Collaborative Machine Learning Method for Heterogeneous Data, Models, and Objectives

Heterogeneous Federated Learning via Grouped Sequential-to-Parallel Training

Hybrid Federated Learning: Algorithms and Implementation.

FedHe: Heterogeneous Models and Communication-Efficient Federated Learning

FedEL: Federated ensemble learning for non-iid data