Abstract:Adaptive moment estimation (Adam), as a Stochastic Gradient Descent (SGD) variant, has gained widespread popularity in federated learning (FL) due to its fast convergence. However, federated Adam (FedAdam) algorithms suffer from a threefold increase in uplink communication overhead compared to federated SGD (FedSGD) algorithms, which arises from the necessity to transmit both local model updates and first and second moment estimates from distributed devices to the centralized server for aggregation. Driven by this issue, we propose a novel sparse FedAdam algorithm called FedAdam-SSM, wherein distributed devices sparsify the updates of local model parameters and moment estimates and subsequently upload the sparse representations to the centralized server. To further reduce the communication overhead, the updates of local model parameters and moment estimates incorporate a shared sparse mask (SSM) into the sparsification process, eliminating the need for three separate sparse masks. Theoretically, we develop an upper bound on the divergence between the local model trained by FedAdam-SSM and the desired model trained by centralized Adam, which is related to sparsification error and imbalanced data distribution. By minimizing the divergence bound between the model trained by FedAdam-SSM and centralized Adam, we optimize the SSM to mitigate the learning performance degradation caused by sparsification error. Additionally, we provide convergence bounds for FedAdam-SSM in both convex and non-convex objective function settings, and investigate the impact of local epoch, learning rate and sparsification ratio on the convergence rate of FedAdam-SSM. Experimental results show that FedAdam-SSM outperforms baselines in terms of convergence rate (over 1.1$\times$ faster than the sparse FedAdam baselines) and test accuracy (over 14.5\% ahead of the quantized FedAdam baselines).

AFedAvg: communication-efficient federated learning aggregation with adaptive communication frequency and gradient sparse

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

AsyncFedED: Asynchronous Federated Learning with Euclidean Distance Based Adaptive Weight Aggregation

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

Communication-Efficient Agnostic Federated Averaging

Towards Communication-efficient Federated Learning via Sparse and Aligned Adaptive Optimization

Communication-Efficient Federated Learning With Data and Client Heterogeneity

FedAgg: Adaptive Federated Learning with Aggregated Gradients

Adaptive Batchsize Selection and Gradient Compression for Wireless Federated Learning

FedPA: An adaptively partial model aggregation strategy in Federated Learning

Robust Federated Averaging via Outlier Pruning

FedDA: Faster Framework of Local Adaptive Gradient Methods via Restarted Dual Averaging

FedLion: Faster Adaptive Federated Optimization with Fewer Communication

Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning

CC-FedAvg: Computationally Customized Federated Averaging

FedPD: A Federated Learning Framework With Adaptivity to Non-IID Data

Toward Communication Efficient Adaptive Gradient Method

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

FedSVRG Based Communication Efficient Scheme for Federated Learning in MEC Networks

Communication Efficient Federated Learning with Adaptive Quantization