Abstract:While petabytes of data are generated each day by a number of independent computing devices, only a few of them can be finally collected and used for deep learning (DL) due to the apprehension of data security and privacy leakage, thus seriously retarding the extension of DL. In such a circumstance, federated learning (FL) was proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning with periodic model averaging (FedAvg) introduced massive communication overhead as the synchronized data in each iteration is about the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant proposals focusing on the communication rounds reduction and data compression were proposed to decrease the communication overhead of FL. In this article, we propose Overlap-FedAvg, an innovative framework that loosed the chain-like constraint of federated learning and paralleled the model training phase with the model communication phase (i.e., uploading local models and downloading the global model), so that the latter phase could be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg was further developed with a hierarchical computing strategy, a data compensation mechanism, and a nesterov accelerated gradients (NAG) algorithm. In Particular, Overlap-FedAvg is orthogonal to many other compression methods so that they could be applied together to maximize the utilization of the cluster. Besides, the theoretical analysis is provided to prove the convergence of the proposed framework. Extensive experiments conducting on both image classification and natural language processing tasks with multiple models and datasets also demonstrate that the proposed framework substantially reduced the communication overhead and boosted the federated learning process.

FedSL: A Communication Efficient Federated Learning with Split Layer Aggregation

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Communication-Efficient Model Aggregation with Layer Divergence Feedback in Federated Learning

FedLite: A Scalable Approach for Federated Learning on Resource-constrained Clients

FedEmb: A Vertical and Hybrid Federated Learning Algorithm using Network And Feature Embedding Aggregation

Communication-and-Computation Efficient Split Federated Learning: Gradient Aggregation and Resource Management

Data-Quality-Driven Federated Learning for Optimizing Communication Costs

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

Communication-Efficient Federated Learning in Channel Constrained Internet of Things

Federated Learning: Strategies for Improving Communication Efficiency

Communication-Efficient Federated Learning with Compensated Overlap-FedAvg

Communication-Efficient Federated Learning: A Variance-Reduced Stochastic Approach with Adaptive Sparsification.

Efficient Model Compression for Hierarchical Federated Learning

Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation

MSRA-Fed:A Communication-Efficient Federated Learning Method Based on Model Split and Representation Aggregate

Boosting Communication Efficiency of Federated Learning's Secure Aggregation

Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination

Communication Efficient Federated Learning With Heterogeneous Structured Client Models

HSFL: an Efficient Split Federated Learning Framework Via Hierarchical Organization.

Efficient and Less Centralized Federated Learning