Abstract:While petabytes of data are generated each day by a number of independent computing devices, only a few of them can be finally collected and used for deep learning (DL) due to the apprehension of data security and privacy leakage, thus seriously retarding the extension of DL. In such a circumstance, federated learning (FL) was proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning with periodic model averaging (FedAvg) introduced massive communication overhead as the synchronized data in each iteration is about the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant proposals focusing on the communication rounds reduction and data compression were proposed to decrease the communication overhead of FL. In this article, we propose Overlap-FedAvg, an innovative framework that loosed the chain-like constraint of federated learning and paralleled the model training phase with the model communication phase (i.e., uploading local models and downloading the global model), so that the latter phase could be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg was further developed with a hierarchical computing strategy, a data compensation mechanism, and a nesterov accelerated gradients (NAG) algorithm. In Particular, Overlap-FedAvg is orthogonal to many other compression methods so that they could be applied together to maximize the utilization of the cluster. Besides, the theoretical analysis is provided to prove the convergence of the proposed framework. Extensive experiments conducting on both image classification and natural language processing tasks with multiple models and datasets also demonstrate that the proposed framework substantially reduced the communication overhead and boosted the federated learning process.

FedComp: A Federated Learning Compression Framework for Resource-Constrained Edge Computing Devices

Deep Learning Based Coded Over-the-Air Computation for Personalized Federated Learning

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

FedTC: Enabling Communication-Efficient Federated Learning Via Transform Coding

FedSC: Compatible Gradient Compression for Communication-Efficient Federated Learning

Secure Federated Learning with Model Compression.

Efficient Federated Learning in Resource-Constrained Edge Intelligence Networks using Model Compression

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Secure Federated Learning over Wireless Communication Networks with Model Compression

FedSZ: Leveraging Error-Bounded Lossy Compression for Federated Learning Communications

Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation

Snowball: Energy Efficient and Accurate Federated Learning with Coarse-to-Fine Compression over Heterogeneous Wireless Edge Devices

Model Compression by Count-Sketch for Over-the-Air Stateless Federated Learning

Federated learning compression designed for lightweight communications

Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing.

Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes

Accelerating Decentralized Federated Learning in Heterogeneous Edge Computing

Efficient Model Compression for Hierarchical Federated Learning

Communication-Efficient Federated Learning with Compensated Overlap-FedAvg