Abstract:While petabytes of data are generated each day by a number of independent computing devices, only a few of them can be finally collected and used for deep learning (DL) due to the apprehension of data security and privacy leakage, thus seriously retarding the extension of DL. In such a circumstance, federated learning (FL) was proposed to perform model training by multiple clients' combined data without the dataset sharing within the cluster. Nevertheless, federated learning with periodic model averaging (FedAvg) introduced massive communication overhead as the synchronized data in each iteration is about the same size as the model, and thereby leading to a low communication efficiency. Consequently, variant proposals focusing on the communication rounds reduction and data compression were proposed to decrease the communication overhead of FL. In this article, we propose Overlap-FedAvg, an innovative framework that loosed the chain-like constraint of federated learning and paralleled the model training phase with the model communication phase (i.e., uploading local models and downloading the global model), so that the latter phase could be totally covered by the former phase. Compared to vanilla FedAvg, Overlap-FedAvg was further developed with a hierarchical computing strategy, a data compensation mechanism, and a nesterov accelerated gradients (NAG) algorithm. In Particular, Overlap-FedAvg is orthogonal to many other compression methods so that they could be applied together to maximize the utilization of the cluster. Besides, the theoretical analysis is provided to prove the convergence of the proposed framework. Extensive experiments conducting on both image classification and natural language processing tasks with multiple models and datasets also demonstrate that the proposed framework substantially reduced the communication overhead and boosted the federated learning process.

FedVAE: Communication-Efficient Federated Learning with Non-IID Private Data

Projected Federated Averaging with Heterogeneous Differential Privacy.

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

VAEFL: Integrating Variational Autoencoders for Privacy Preservation and Performance Retention in Federated Learning

Fed2VAEs: an Efficient Privacy-Preserving Federated Learning Approach Based on Variational Autoencoders

Communication-Efficient Federated Data Augmentation on Non-IID Data

Optimizing Federated Learning on Non-IID Data Using Local Shapley Value.

Sparse Federated Learning with Hierarchical Personalized Models

Communication-Efficient Federated Learning with Compensated Overlap-FedAvg

Sparse Federated Learning with Hierarchical Personalization Models.

A Communication Efficient Vertical Federated Learning Framework.

Communication-Efficient and Privacy-Preserving Aggregation in Federated Learning With Adaptability

FedVeca: Federated Vectorized Averaging on Non-IID Data with Adaptive Bi-directional Global Objective

FedFC: an Efficient Personalized Federated Learning Method on Non-iid Data

Improving Communication Efficiency: A Dual Adaptive Federated Learning Framework

FedV: Privacy-Preserving Federated Learning over Vertically Partitioned Data

Federated Learning via Variational Bayesian Inference: Personalization, Sparsity and Clustering

TVFL: Tunable Vertical Federated Learning Towards Communication-Efficient Model Serving.

FedVAE: Trajectory privacy preserving based on Federated Variational AutoEncoder

EFVAE: Efficient Federated Variational Autoencoder for Collaborative Filtering

FedKG: Model-Optimized Federated Learning for Local Client Training with Non-IID Private Data