Abstract:Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, these methods encounter challenges such as the straggler problem and diminished model performance due to heterogeneous bandwidth and non-IID (Independently and Identically Distributed) data. To address these issues, we introduce a bandwidth-aware compression framework for FL, aimed at improving communication efficiency while mitigating the problems associated with non-IID data. First, our strategy dynamically adjusts compression ratios according to bandwidth, enabling clients to upload their models at a close pace, thus exploiting the otherwise wasted time to transmit more data. Second, we identify the non-overlapped pattern of retained parameters after compression, which results in diminished client update signals due to uniformly averaged weights. Based on this finding, we propose a parameter mask to adjust the client-averaging coefficients at the parameter level, thereby more closely approximating the original updates, and improving the training convergence under heterogeneous environments. Our evaluations reveal that our method significantly boosts model accuracy, with a maximum improvement of 13% over the uncompressed FedAvg. Moreover, it achieves a $3.37\times$ speedup in reaching the target accuracy compared to FedAvg with a Top-K compressor, demonstrating its effectiveness in accelerating convergence with compression. The integration of common compression techniques into our framework further establishes its potential as a versatile foundation for future cross-device, communication-efficient FL research, addressing critical challenges in FL and advancing the field of distributed machine learning.

Communication-efficient Federated Learning Through 1-Bit Compressive Sensing and Analog Aggregation.

1-Bit Compressive Sensing for Efficient Federated Learning Over the Air

Stochastic Controlled Averaging for Federated Learning with Communication Compression

Joint Optimization for Federated Learning over the Air

Communication-efficient federated learning based on compressed sensing and ternary quantization

Joint Optimization of Communications and Federated Learning over the Air

Communication and Computation Efficient Federated Learning for Internet of Vehicles with a Constrained Latency

Secure Federated Learning over Wireless Communication Networks with Model Compression

Over-the-Air Federated Learning with Compressed Sensing: Is Sparsification Necessary?

Communication-Efficient and Secure Federated Learning Based on Adaptive One-Bit Compressed Sensing

Communication-Efficient Federated Learning via Quantized Compressed Sensing

Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing.

Energy and Spectrum Efficient Federated Learning via High-Precision Over-the-Air Computation

Secure Federated Learning with Model Compression.

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

Message Passing Based Wireless Federated Learning Via Analog Message Aggregation

Model Compression by Count-Sketch for Over-the-Air Stateless Federated Learning

Federated Learning via Intelligent Reflecting Surface

Over-the-Air Federated Learning and Optimization

Latency-Efficient Wireless Federated Learning with Spasification and Quantization for Heterogeneous Devices

Digital versus Analog Transmissions for Federated Learning over Wireless Networks