Abstract:Federated learning obtains the global model through the cooperative training of each participating device while protecting data privacy. However, the huge communication cost caused by each client sending complete model update becomes an important problem for its wide application. Model sparsification and quantization are practical solutions to reduce single-round uplink communication, respectively. However, when the compression ratio increases, (1) the current model sparsification methods based on fine granularity by uploading significant gradients are difficult to effectively select important parameters by using thresholds; (2) the unified bit-width quantization of parameters of layers with different importance in the model will lead to greater information loss. In this paper, we propose a joint compression framework based on functional structure sparsity and hybrid quantization sensing to compress the communication cost to more than two orders of magnitude. First, we proposed to take filters as sparse granularity through the visual analysis of parameter updating rules in the training process, and searched for filters with strong representational ability among filters with different representational abilities in the convolution layer. Second, we propose an adaptive layer-sensing quantization method according to the parameter distribution of each layer, which assigns different bit widths to different layers in the model after structural sparsification. Finally, we design a model aggregation and update scheme based on the above joint compression framework, which reduces the error caused by compression through model reconstruction and parameter reuse. Experiments show that our proposed framework compress single communication to more than two orders of magnitude in different tasks while ensuring the convergence speed and final model performance.

Optimized Compressed Sensing for Communication Efficient Federated Learning.

Efficient Privacy-Preserving Federated Learning with Improved Compressed Sensing

Communication Efficient Federated Learning Based on Combination of Structural Sparsification and Hybrid Quantization Sensing

Communication-Efficient and Secure Federated Learning Based on Adaptive One-Bit Compressed Sensing

Communication-Efficient Federated Learning via Quantized Compressed Sensing

A Novel Adaptive Gradient Compression Approach for Communication-Efficient Federated Learning

Communication-Efficient Federated Learning Based on Compressed Sensing

Communication-Efficient Federated Learning Based on Chained Secure Multiparty Computing and Compressed Sensing.

FL-OTCSEnc: Towards Secure Federated Learning with Deep Compressed Sensing

A Compressive Sensing Approach for Federated Learning Over Massive MIMO Communication Systems

Federated Optimization Based on Compression and Event-triggered Communication

Communication-Efficient Federated Learning with Adaptive Compression under Dynamic Bandwidth

Efficient Model Compression for Hierarchical Federated Learning

Secure Federated Learning with Model Compression.

Communication-Efficient Federated Learning with Sparsity and Quantization

Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing.

Communication-efficient federated learning based on compressed sensing and ternary quantization

Ternary Compression for Communication-Efficient Federated Learning.

Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence

Secure Model-Contrastive Federated Learning With Improved Compressive Sensing