Abstract:Federated learning (FL) is a distributed machine learning approach that allows multiple devices or computing nodes to jointly train models without sharing raw data. However, in real-world application scenarios, FL usually encounters a critical challenge of data heterogeneity. Recent studies have revealed that the client’s model suffers severe bias between the local model and global model, leading to global performance degradation. Improving the generalization of local learning would inherently reduce bias. It has been proved that self-paced learning on a single device can greatly achieve a better generalization result. However, it is not well explored how it can be applied to federated learning with a number of distributed nodes working cooperatively. Specifically, self-paced learning suggests using easy data and then gradually difficult data during model training. It is not straightforward to differentiate “easy” and “difficult” data at the local since global data distribution is not available, especially with severe data heterogeneity. To address the above issues, we propose a novel federated learning framework, Federation-Paced Learning (FedPL), which enables a self-paced process in federated learning and effectively improves the model performance. First, we propose schemes to analyze the data characteristics in terms of difficulty. Then we define a stage controller to synchronize the learning process across cooperative nodes to follow the easy-to-hard rule. Finally, we propose a client selection strategy to further improve the learning efficacy. We evaluate the performance of FedPL on several generic public datasets. Experiment results show that the proposed FedPL outperforms existing methods by up to 13.50% in terms of accuracy. Code is available at https://github.com/tnghua/FedPL.

Pipelined Training and Transmission for Federated Learning

ProgFed: Effective, Communication, and Computation Efficient Federated Learning by Progressive Training

Federation-Paced Learning: Towards Efficient Federated Learning with Synchronized Pace

FedACT: an Adaptive Chained Training Approach for Federated Learning in Computing Power Networks

Accelerating Federated Learning with Data and Model Parallelism in Edge Computing

Adaptive Control of Local Updating and Model Compression for Efficient Federated Learning

Accelerating Federated Learning with Cluster Construction and Hierarchical Aggregation.

Accelerating Federated Learning with Model Segmentation for Edge Networks

FedLC: Accelerating Asynchronous Federated Learning in Edge Computing

No One Idles: Efficient Heterogeneous Federated Learning with Parallel Edge and Server Computation.

FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Framework for Heterogeneous Edge Devices

FedPE: Adaptive Model Pruning-Expanding for Federated Learning on Mobile Devices

PedFed: A performance evaluation-driven federated learning framework for efficient communication

FTPipeHD: A Fault-Tolerant Pipeline-Parallel Distributed Training Approach for Heterogeneous Edge Devices

EdgeFed: Optimized Federated Learning Based on Edge Computing

Adaptive Configuration for Heterogeneous Participants in Decentralized Federated Learning

An Efficient Asynchronous Federated Learning Protocol for Edge Devices

FedStar: Efficient Federated Learning on Heterogeneous Communication Networks

FedPCC: Parallelism of Communication and Computation for Federated Learning in Wireless Networks

FedEdge: Accelerating Edge-Assisted Federated Learning

Adaptive device sampling and deadline determination for cloud-based heterogeneous federated learning