Abstract:Federated learning obtains the global model through the cooperative training of each participating device while protecting data privacy. However, the huge communication cost caused by each client sending complete model update becomes an important problem for its wide application. Model sparsification and quantization are practical solutions to reduce single-round uplink communication, respectively. However, when the compression ratio increases, (1) the current model sparsification methods based on fine granularity by uploading significant gradients are difficult to effectively select important parameters by using thresholds; (2) the unified bit-width quantization of parameters of layers with different importance in the model will lead to greater information loss. In this paper, we propose a joint compression framework based on functional structure sparsity and hybrid quantization sensing to compress the communication cost to more than two orders of magnitude. First, we proposed to take filters as sparse granularity through the visual analysis of parameter updating rules in the training process, and searched for filters with strong representational ability among filters with different representational abilities in the convolution layer. Second, we propose an adaptive layer-sensing quantization method according to the parameter distribution of each layer, which assigns different bit widths to different layers in the model after structural sparsification. Finally, we design a model aggregation and update scheme based on the above joint compression framework, which reduces the error caused by compression through model reconstruction and parameter reuse. Experiments show that our proposed framework compress single communication to more than two orders of magnitude in different tasks while ensuring the convergence speed and final model performance.

Joint Sparsification and Quantization for Wireless Federated Learning under Communication Constraints

Joint Sparsification and Quantization for Heterogeneous Devices in Energy Constrained Federated Learning

Latency-Efficient Wireless Federated Learning with Spasification and Quantization for Heterogeneous Devices

Communication Efficient Federated Learning Based on Combination of Structural Sparsification and Hybrid Quantization Sensing

Communication-Efficient Federated Learning with Sparsity and Quantization

Secure Federated Learning over Wireless Communication Networks with Model Compression

Secure Federated Learning with Model Compression.

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Communication-Efficient Federated Learning via Quantized Compressed Sensing

Quantization Bits Allocation for Wireless Federated Learning

Joint Optimization of Communications and Federated Learning over the Air

Adaptive Network Pruning for Wireless Federated Learning

Communication-efficient Federated Learning Through 1-Bit Compressive Sensing and Analog Aggregation.

A Joint Learning and Communications Framework for Federated Learning over Wireless Networks

FedVQCS: Federated Learning via Vector Quantized Compressed Sensing

Communication-efficient federated learning based on compressed sensing and ternary quantization

JointSQ: Joint Sparsification-Quantization for Distributed Learning

Joint Optimization for Federated Learning over the Air

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

Joint Compression and Deadline Optimization for Wireless Federated Learning

Stochastic gradient compression for federated learning over wireless network