Abstract:With the growth of participating clients, the centralized parameter server (PS) will seriously limit the scale and efficiency of Federated Learning (FL). A straightforward approach to scale up the FL system is to construct a Parallel FL (PFL) system with multiple parallel PSes. However, it is unclear whether PFL can really accelerate FL or reduce the training time of FL. Even if the answer is yes, it is non-trivial to design a highly efficient parameter average algorithm for a PFL system. In this paper, we propose a completely parallelizable FL algorithm called P-FedAvg under the PFL architecture. P-FedAvg extends the well-known FedAvg algorithm by allowing multiple PSes to cooperate and train a learning model together. In P-FedAvg, each PS is only responsible for a fraction of total clients, but PSes can mix model parameters in a dedicatedly designed way so that the FL model can well converge. Different from heuristic-based algorithms, P-FedAvg is with theoretical guarantees. To be rigorous, we theoretically analyze the convergence rate of P-FedAvg in terms of the number of conducted iterations, the communication cost of each global iteration and the optimal weights for each PS to mix parameters with its neighbors. Based on theoretical analysis, we conduct a case study on five typical overlay topolgoies formed by PSes to further examine the communication efficiency under different topologies, and investigate how the overlay topology affects the convergence rate, communication cost and robustness of a PFL system. Lastly, we perform extensive experiments with real datasets to verify our analysis and demonstrate that P-FedAvg can significantly speed up FL than traditional FedAvg and other competitive baselines. We believe that our work can help to lay a theoretical foundation for building more efficient PFL systems.

Federated Primal Dual Fixed Point Algorithm

FedDGP: Disentangling Global and Personal Models for Federated Learning

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

A-FedPD: Aligning Dual-Drift is All Federated Primal-Dual Learning Needs

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

DualFL: A Duality-based Federated Learning Algorithm with Communication Acceleration in the General Convex Regime

Privacy-preserving Federated Primal-dual Learning for Non-convex and Non-smooth Problems with Model Sparsification

Federated Composite Saddle Point Optimization

Multi-consensus decentralized primal-dual fixed point algorithm for distributed learning

FedPD: A Federated Learning Framework with Optimal Rates and Adaptivity to Non-IID Data.

FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data

Exact Penalty Method for Federated Learning

Federated Frank-Wolfe Algorithm

Preconditioned Federated Learning

Two Modified Schemes for the Primal Dual Fixed Point Method

Probably Approximately Correct Federated Learning

Resource-constrained Federated Learning with Heterogeneous Data: Formulation and Analysis

Accelerating Federated Learning Via Parallel Servers: A Theoretically Guaranteed Approach.

Federated Optimization Via Knowledge Codistillation.

Clustered Federated Learning based on Nonconvex Pairwise Fusion

FedFDP: Fairness-Aware Federated Learning with Differential Privacy