Abstract:Top-K sparsification-based compression techniques are popular and powerful for reducing communication costs in federated learning (FL). However, existing Top-K sparsification-based compression methods suffer from two critical issues that severely hinder their implementation, particularly in the context of FL, which often involves a vast number of resource-constrained devices: 1) the low compressibility of the Top-K parameter’s indexes significantly limits the overall compression ratio (CR) and 2) the residual accumulation techniques used to maintain the model quality consume huge memory resources. To address these issues, we propose a novel FL compression framework, named FedComp, for deep neural networks (DNNs). FedComp achieves a higher communication CR while maintaining comparable model quality at low memory cost. Specifically, FedComp incorporates the following three key components: 1) a tensor-wise index-sharing mechanism that greatly reduces the index proportion by sharing one index among multiple elements of the tensor; 2) a fine-grained parameters packing strategy that reduces the transmission of duplicate value and index by considering their properties, thereby further reducing the overall communication cost; and 3) a residual compressor that significantly reduces memory cost by enhancing the compressibility of floating-point residuals and achieving a high CR with a lossless encoding scheme. Experiments on mainstream machine learning (ML) tasks with different DNN structures and datasets demonstrate that our proposed FedComp outperforms the state-of-the-art FL compression algorithms by achieving a higher communication CR of up to $28.5\times $ while reducing memory costs by $21.04\times $ – $50.59\times $ on the local residual model, without degrading FL training performance.

Snowball: Energy Efficient and Accurate Federated Learning with Coarse-to-Fine Compression over Heterogeneous Wireless Edge Devices

Snowball Effect in Federated Learning: an Approach of Exponentially Expanding Structures for Optimizing the Training Efficiency

Secure Federated Learning with Model Compression.

Secure Federated Learning over Wireless Communication Networks with Model Compression

To Talk or to Work: Flexible Communication Compression for Energy Efficient Federated Learning over Heterogeneous Mobile Edge Devices

Adaptive Batchsize Selection and Gradient Compression for Wireless Federated Learning

Latency-Efficient Wireless Federated Learning with Spasification and Quantization for Heterogeneous Devices

Efficient Federated Learning in Resource-Constrained Edge Intelligence Networks using Model Compression

Over-the-Air Federated Learning and Optimization

Efficient Model Compression for Hierarchical Federated Learning

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

Stochastic gradient compression for federated learning over wireless network

Joint Sparsification and Quantization for Heterogeneous Devices in Energy Constrained Federated Learning

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.

Federated Learning With Client Selection and Gradient Compression in Heterogeneous Edge Systems

Energy Efficient Federated Learning over Heterogeneous Mobile Devices via Joint Design of Weight Quantization and Wireless Transmission

FedComp: A Federated Learning Compression Framework for Resource-Constrained Edge Computing Devices

Heterogeneity-Aware Cooperative Federated Edge Learning with Adaptive Computation and Communication Compression

AnycostFL: Efficient On-Demand Federated Learning over Heterogeneous Edge Devices

Accelerating Decentralized Federated Learning in Heterogeneous Edge Computing

Adaptive Control of Local Updating and Model Compression for Efficient Federated Learning