Abstract:Federated learning (FL) commonly encourages the clients to perform multiple local updates before the global aggregation, thus avoiding frequent model exchanges and relieving the communication bottleneck between the server and clients. Though empirically effective, the negative impact of multiple local updates on the stability of FL is not thoroughly studied, which may result in a globally unstable and slow convergence. Based on sensitivity analysis, we define in this paper a local-update stability index for the general FL, as measured by the maximum inter-client model discrepancy after the multiple local updates that mainly stems from the data heterogeneity. It enables to determine how much the variation of client's models with multiple local updates may influence the global model, and can also be linked with the convergence and generalization. We theoretically derive the proposed local-update stability for current state-of-the-art FL methods, providing possible insight to understanding their motivation and limitation from a new perspective of stability. For example, naively executing the parallel acceleration locally at clients would harm the local-update stability. Motivated by this, we then propose a novel accelerated yet stabilized FL algorithm (named FedANAG) based on the server- and client-level Nesterov accelerated gradient (NAG). In FedANAG, the global and local momenta are elaborately designed and alternatively updated, while the stability of local update is enhanced with help of the global momentum. We prove the convergence of FedANAG for strongly convex, general convex and non-convex settings. We then conduct evaluations on both the synthetic and real-world datasets to first validate our proposed local-update stability. The results further show that across various data heterogeneity and client participation ratios, FedANAG not only accelerates the global convergence by reducing the required number of communication rounds to a target accuracy, but converges to an eventually higher accuracy.

HybridCom: Improve Federated Learning Efficiency on Unstable Data

Accelerating Hybrid Federated Learning Convergence under Partial Participation

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

FedCom: A Byzantine-Robust Local Model Aggregation Rule Using Data Commitment for Federated Learning

HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clients

Fast Composite Optimization and Statistical Recovery in Federated Learning

Efficient and Less Centralized Federated Learning

ShapleyFL: Robust Federated Learning Based on Shapley Value

Federated Learning via Consensus Mechanism on Heterogeneous Data: A New Perspective on Convergence

Efficient Model Compression for Hierarchical Federated Learning

Stabilizing and Accelerating Federated Learning on Heterogeneous Data With Partial Client Participation

Preconditioned Federated Learning

FedConv: A Learning-on-Model Paradigm for Heterogeneous Federated Clients

Balancing Similarity and Complementarity for Federated Learning

Optimizing Hierarchical Federated Learning: A Reinforcement Learning Approach

FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement

Communication-Efficient Federated Learning with Adaptive Compression under Dynamic Bandwidth

Flexible Clustered Federated Learning for Client-Level Data Distribution Shift

Aligning Model Outputs for Class Imbalanced Non-Iid Federated Learning

Compositional federated learning: Applications in distributionally robust averaging and meta learning

Towards Efficient Federated Learning Framework Via Selective Aggregation of Models