Abstract:While petabytes of data are generated each day by a number of independent computing devices, only a few of them can be finally collected and used for deep learning (DL) due to the apprehension of data security and privacy leakage, thus seriously retarding the extension of DL. In such a circumstance, federated learning (FL) was proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning with periodic model averaging (FedAvg) introduced massive communication overhead as the synchronized data in each iteration is about the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant proposals focusing on the communication rounds reduction and data compression were proposed to decrease the communication overhead of FL. In this article, we propose Overlap-FedAvg, an innovative framework that loosed the chain-like constraint of federated learning and paralleled the model training phase with the model communication phase (i.e., uploading local models and downloading the global model), so that the latter phase could be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg was further developed with a hierarchical computing strategy, a data compensation mechanism, and a nesterov accelerated gradients (NAG) algorithm. In Particular, Overlap-FedAvg is orthogonal to many other compression methods so that they could be applied together to maximize the utilization of the cluster. Besides, the theoretical analysis is provided to prove the convergence of the proposed framework. Extensive experiments conducting on both image classification and natural language processing tasks with multiple models and datasets also demonstrate that the proposed framework substantially reduced the communication overhead and boosted the federated learning process.

FedBCGD: Communication-Efficient Accelerated Block Coordinate Gradient Descent for Federated Learning

FedDGP: Disentangling Global and Personal Models for Federated Learning

Lazily Aggregated Quantized Gradient Innovation for Communication-Efficient Federated Learning.

Save It All: Enabling Full Parameter Tuning for Federated Large Language Models via Cycle Block Gradient Descent

Adaptive Control of Client Selection and Gradient Compression for Efficient Federated Learning

Adaptive Block-Wise Regularization and Knowledge Distillation for Enhancing Federated Learning

Federated Learning Method Based on Knowledge Distillation and Deep Gradient Compression

A Communication Efficient Vertical Federated Learning Framework.

FedDD: Toward Communication-efficient Federated Learning with Differential Parameter Dropout

Communication-Efficient Federated Learning with Accelerated Client Gradient

FedBCD: A Communication-Efficient Collaborative Learning Framework for Distributed Features

Communication-Efficient Model Aggregation with Layer Divergence Feedback in Federated Learning

Conquering the Communication Constraints to Enable Large Pre-Trained Models in Federated Learning

Communication-Efficient Federated Learning with Compensated Overlap-FedAvg

A Novel Adaptive Gradient Compression Approach for Communication-Efficient Federated Learning

AFedAvg: communication-efficient federated learning aggregation with adaptive communication frequency and gradient sparse

Communication-and-Computation Efficient Split Federated Learning: Gradient Aggregation and Resource Management

Toward Communication Efficient Adaptive Gradient Method

FedSL: A Communication Efficient Federated Learning with Split Layer Aggregation

Adaptive Federated Dropout: Improving Communication Efficiency and Generalization for Federated Learning

Communication-Efficient Federated Learning: A Variance-Reduced Stochastic Approach with Adaptive Sparsification.