Abstract:Adaptive moment estimation (Adam), as a Stochastic Gradient Descent (SGD) variant, has gained widespread popularity in federated learning (FL) due to its fast convergence. However, federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead compared to federated SGD (FedSGD) algorithms, which arises from the necessity to transmit both local model updates and first and second moment estimates from distributed devices to the centralized server for aggregation. Driven by this issue, we propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates of local model parameters and moment estimates and subsequently upload the sparse representations to the centralized server. To further reduce the communication overhead, the updates of local model parameters and moment estimates incorporate a shared sparse mask (SSM) into the sparsification process, eliminating the need for three separate sparse masks. Theoretically, we develop an upper bound on the divergence between the local model trained by FedAdam-SSM and the desired model trained by centralized Adam, which is related to sparsification error and imbalanced data distribution. By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error. Additionally, we provide convergence bounds for FedAdam-SSM in both convex and non-convex objective function settings, and investigate the impact of local epoch, learning rate and sparsification ratio on the convergence rate of FedAdam-SSM. Experimental results show that FedAdam-SSM outperforms baselines in terms of convergence rate (over 1.1$\times$ faster than the sparse FedAdam baselines) and test accuracy (over 14.5\% ahead of the quantized FedAdam baselines).

FED-SDS: Adaptive Structured Dynamic Sparsity for Federated Learning under Heterogeneous Clients

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Federated learning with workload-aware client scheduling in heterogeneous systems

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better

Efficient Personalized Federated Learning via Sparse Model-Adaptation

Federated learning based on stratified sampling and regularization

Federated Dynamic Client Selection for Fairness Guarantee in Heterogeneous Edge Computing

FedSW: Federated learning with adaptive sample weights

Federated Learning Using Sparse-Adaptive Model Selection for Embedded Edge Computing

Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization

An Aggregation-Free Federated Learning for Tackling Data Heterogeneity

FedSPD: A Soft-clustering Approach for Personalized Decentralized Federated Learning

Decentralized Sporadic Federated Learning: A Unified Algorithmic Framework with Convergence Guarantees

CDFed: Contribution-based Dynamic Federated Learning for Managing System and Statistical Heterogeneity

Achieving Personalized Federated Learning with Sparse Local Models

Accelerating Federated Learning via Sequential Training of Grouped Heterogeneous Clients

FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learning

FedEP: Tailoring Attention to Heterogeneous Data Distribution with Entropy Pooling for Decentralized Federated Learning

Towards addressing aggregation deviation for model training in resource-scarce edge environment

FedADMM: A Robust Federated Deep Learning Framework with Adaptivity to System Heterogeneity

Tackling Mavericks in Federated Learning via Adaptive Client Selection Strategy