Abstract:Federated learning (FL) commonly encourages the clients to perform multiple local updates before the global aggregation, thus avoiding frequent model exchanges and relieving the communication bottleneck between the server and clients. Though empirically effective, the negative impact of multiple local updates on the stability of FL is not thoroughly studied, which may result in a globally unstable and slow convergence. Based on sensitivity analysis, we define in this paper a local-update stability index for the general FL, as measured by the maximum inter-client model discrepancy after the multiple local updates that mainly stems from the data heterogeneity. It enables to determine how much the variation of client's models with multiple local updates may influence the global model, and can also be linked with the convergence and generalization. We theoretically derive the proposed local-update stability for current state-of-the-art FL methods, providing possible insight to understanding their motivation and limitation from a new perspective of stability. For example, naively executing the parallel acceleration locally at clients would harm the local-update stability. Motivated by this, we then propose a novel accelerated yet stabilized FL algorithm (named FedANAG) based on the server- and client-level Nesterov accelerated gradient (NAG). In FedANAG, the global and local momenta are elaborately designed and alternatively updated, while the stability of local update is enhanced with help of the global momentum. We prove the convergence of FedANAG for strongly convex, general convex and non-convex settings. We then conduct evaluations on both the synthetic and real-world datasets to first validate our proposed local-update stability. The results further show that across various data heterogeneity and client participation ratios, FedANAG not only accelerates the global convergence by reducing the required number of communication rounds to a target accuracy, but converges to an eventually higher accuracy.

FedMoS: Taming Client Drift in Federated Learning with Double Momentum and Adaptive Selection.

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Momentum Benefits Non-iid Federated Learning Simply and Provably

Communication-Efficient Federated Learning with Acceleration of Global Momentum

FedCM: Federated Learning with Client-level Momentum

Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression.

Adaptive Control of Client Selection and Gradient Compression for Efficient Federated Learning

Enhance Local Consistency for Free: A Multi-Step Inertial Momentum Approach

Improving Communication Efficiency: A Dual Adaptive Federated Learning Framework

FedCo: Self-Supervised Learning in Federated Learning with Momentum Contrast.

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

Efficient Federated Learning via Local Adaptive Amended Optimizer with Linear Speedup

Faster Non-Convex Federated Learning via Global and Local Momentum

Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization

Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection

Fairness and Accuracy in Federated Learning

Efficient Federated Learning Using Dynamic Update and Adaptive Pruning with Momentum on Shared Server Data

Stabilizing and Accelerating Federated Learning on Heterogeneous Data With Partial Client Participation

FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement