Abstract:Federated learning (FL) is a distributed machine learning approach that allows multiple devices or computing nodes to jointly train models without sharing raw data. However, in real-world application scenarios, FL usually encounters a critical challenge of data heterogeneity. Recent studies have revealed that the client’s model suffers severe bias between the local model and global model, leading to global performance degradation. Improving the generalization of local learning would inherently reduce bias. It has been proved that self-paced learning on a single device can greatly achieve a better generalization result. However, it is not well explored how it can be applied to federated learning with a number of distributed nodes working cooperatively. Specifically, self-paced learning suggests using easy data and then gradually difficult data during model training. It is not straightforward to differentiate “easy” and “difficult” data at the local since global data distribution is not available, especially with severe data heterogeneity. To address the above issues, we propose a novel federated learning framework, Federation-Paced Learning (FedPL), which enables a self-paced process in federated learning and effectively improves the model performance. First, we propose schemes to analyze the data characteristics in terms of difficulty. Then we define a stage controller to synchronize the learning process across cooperative nodes to follow the easy-to-hard rule. Finally, we propose a client selection strategy to further improve the learning efficacy. We evaluate the performance of FedPL on several generic public datasets. Experiment results show that the proposed FedPL outperforms existing methods by up to 13.50% in terms of accuracy. Code is available at https://github.com/tnghua/FedPL.

Learning by Imitating the Classics: Mitigating Class Imbalance in Federated Learning Via Simulated Centralized Learning

FedSC: A federated learning algorithm based on client-side clustering

Accelerating Hybrid Federated Learning Convergence under Partial Participation

FedCML: Federated Clustering Mutual Learning with non-IID Data.

FFEDCL: Fair Federated Learning with Contrastive Learning

Clients Help Clients: Alternating Collaboration for Semi-Supervised Federated Learning

FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learning

A Survey on Class Imbalance in Federated Learning

Rethinking Client Drift in Federated Learning: A Logit Perspective

FedReMa: Improving Personalized Federated Learning via Leveraging the Most Relevant Clients

FedBalancer: Data and Pace Control for Efficient Federated Learning on Heterogeneous Clients

FedGroup: Accurate Federated Learning via Decomposed Similarity-Based Clustering

Federated Class-Incremental Learning with New-Class Augmented Self-Distillation

Towards Instance-adaptive Inference for Federated Learning

FedClust: Tackling Data Heterogeneity in Federated Learning through Weight-Driven Client Clustering

Federation-Paced Learning: Towards Efficient Federated Learning with Synchronized Pace

Flexible Clustered Federated Learning for Client-Level Data Distribution Shift

MimiC: Combating Client Dropouts in Federated Learning by Mimicking Central Updates

An Experimental Study of Class Imbalance in Federated Learning

FedMeS: Personalized Federated Continual Learning Leveraging Local Memory

A Clustering Federated Learning Algorithm Based on Three-Layer Structure