Abstract:With the growth of participating clients, the centralized parameter server (PS) will seriously limit the scale and efficiency of Federated Learning (FL). A straightforward approach to scale up the FL system is to construct a Parallel FL (PFL) system with multiple parallel PSes. However, it is unclear whether PFL can really accelerate FL or reduce the training time of FL. Even if the answer is yes, it is non-trivial to design a highly efficient parameter average algorithm for a PFL system. In this paper, we propose a completely parallelizable FL algorithm called P-FedAvg under the PFL architecture. P-FedAvg extends the well-known FedAvg algorithm by allowing multiple PSes to cooperate and train a learning model together. In P-FedAvg, each PS is only responsible for a fraction of total clients, but PSes can mix model parameters in a dedicatedly designed way so that the FL model can well converge. Different from heuristic-based algorithms, P-FedAvg is with theoretical guarantees. To be rigorous, we theoretically analyze the convergence rate of P-FedAvg in terms of the number of conducted iterations, the communication cost of each global iteration and the optimal weights for each PS to mix parameters with its neighbors. Based on theoretical analysis, we conduct a case study on five typical overlay topolgoies formed by PSes to further examine the communication efficiency under different topologies, and investigate how the overlay topology affects the convergence rate, communication cost and robustness of a PFL system. Lastly, we perform extensive experiments with real datasets to verify our analysis and demonstrate that P-FedAvg can significantly speed up FL than traditional FedAvg and other competitive baselines. We believe that our work can help to lay a theoretical foundation for building more efficient PFL systems.

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

FedPE: Adaptive Model Pruning-Expanding for Federated Learning on Mobile Devices

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

FedAPEN: Personalized Cross-silo Federated Learning with Adaptability to Statistical Heterogeneity

Efficient Personalized Federated Learning via Sparse Model-Adaptation

Achieving Personalized Federated Learning with Sparse Local Models

FedPA: An adaptively partial model aggregation strategy in Federated Learning

Pfedsim: Similarity-Aware Model Aggregation Towards Personalized Federated Learning.

pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharing

Sparse Federated Learning with Hierarchical Personalization Models.

Accelerating Federated Learning Via Parallel Servers: A Theoretically Guaranteed Approach.

Sparse Federated Learning with Hierarchical Personalized Models

FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learning

No One Idles: Efficient Heterogeneous Federated Learning with Parallel Edge and Server Computation.

FedFC: an Efficient Personalized Federated Learning Method on Non-iid Data

Es-Fedavg: Early-Exit Personalized Federated Learning with Sparse Weight for Adaptive Computation

Sparse Personalized Federated Learning

Efficient and Straggler-Resistant Homomorphic Encryption for Heterogeneous Federated Learning

FedFed: Feature Distillation Against Data Heterogeneity in Federated Learning

FED-SDS: Adaptive Structured Dynamic Sparsity for Federated Learning under Heterogeneous Clients

Feed: Towards Personalization-Effective Federated Learning