Abstract:As a typical privacy-aware machine learning paradigm, federated learning (FL) provides facilities to individually train edge clients with their private data and aggregate the central global model. In this way, privacy leakage can be prevented. Massive communication overhead caused by exchanging updated weights between clients and the server is one of the main obstacles in this strategy. Prior work advocates compressing the weights by employing quantization, gradient sparsification, and knowledge distillation approaches. However, most of them cannot be readily applied to secure aggregation in privacy-aware FL. Some research has made great progress in directly utilizing benchmark secure aggregation protocols on top of the non-privacy-aware FL. Graph-based and gradient-based sparsification has been widely adopted in previous studies. However, the results of reducing communication costs are still unsatisfactory. In this paper, we present a novel communication-efficient privacy-ware FL algorithm from a distinct perspective. We design a new Two-Phase Sparsification with Secure Aggregation (TPSSA) algorithm. In the subnetwork phase, we identify sparse subnetworks by freezing the initial random weights in sufficiently overparametrized networks. All edge clients collaboratively train to discover their subnetwork inside a dense randomly weighted neural network. Then the server aggregates to compute the global model. In the gradient phase, for each pair of edge clients, we introduce pairwise multiplicative random masks to identify the sparsification pattern. Then updates from surviving clients can be correctly cancelled out during the aggregation process in the server. Theoretical analysis reveals convergence, privacy and performance guarantee. We show improvements in accuracy, communication, and computation over traditional and sparsified secure aggregation benchmarks on two real-world datasets.

Time-Correlated Sparsification for Communication-Efficient Federated Learning

FedPSE: Personalized Sparsification with Element-wise Aggregation for Federated Learning

Time-Correlated Sparsification for Efficient Over-the-Air Model Aggregation in Wireless Federated Learning

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Sparse Training for Federated Learning with Regularized Error Correction

Communication-Efficient Federated Learning: A Variance-Reduced Stochastic Approach with Adaptive Sparsification.

SAFARI: Sparsity enabled Federated Learning with Limited and Unreliable Communications

SpaFL: Communication-Efficient Federated Learning with Sparse Models and Low computational Overhead

Over-the-Air Federated Learning with Compressed Sensing: Is Sparsification Necessary?

Communication-Efficient Federated Learning Based on Chained Secure Multiparty Computing and Compressed Sensing.

Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing.

Complement Sparsification: Low-Overhead Model Pruning for Federated Learning

DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training

Communication-Efficient Federated Learning in Channel Constrained Internet of Things

Two-Phase Sparsification with Secure Aggregation for Privacy-Aware Federated Learning

Communication Efficient Federated Learning Based on Combination of Structural Sparsification and Hybrid Quantization Sensing

Communication-Efficient Federated Learning via Regularized Sparse Random Networks

FLCP: federated learning framework with communication-efficient and privacy-preserving

Timely Communication in Federated Learning

Federated Dynamic Sparse Training: Computing Less, Communicating Less, Yet Learning Better