Abstract:Current data compression methods, such as sparsification in Federated Averaging (FedAvg), effectively enhance the communication efficiency of Federated Learning (FL). However, these methods encounter challenges such as the straggler problem and diminished model performance due to heterogeneous bandwidth and non-IID (Independently and Identically Distributed) data. To address these issues, we introduce a bandwidth-aware compression framework for FL, aimed at improving communication efficiency while mitigating the problems associated with non-IID data. First, our strategy dynamically adjusts compression ratios according to bandwidth, enabling clients to upload their models at a close pace, thus exploiting the otherwise wasted time to transmit more data. Second, we identify the non-overlapped pattern of retained parameters after compression, which results in diminished client update signals due to uniformly averaged weights. Based on this finding, we propose a parameter mask to adjust the client-averaging coefficients at the parameter level, thereby more closely approximating the original updates, and improving the training convergence under heterogeneous environments. Our evaluations reveal that our method significantly boosts model accuracy, with a maximum improvement of 13% over the uncompressed FedAvg. Moreover, it achieves a $3.37\times$ speedup in reaching the target accuracy compared to FedAvg with a Top-K compressor, demonstrating its effectiveness in accelerating convergence with compression. The integration of common compression techniques into our framework further establishes its potential as a versatile foundation for future cross-device, communication-efficient FL research, addressing critical challenges in FL and advancing the field of distributed machine learning.

FedMDC: Enabling Communication-Efficient Federated Learning over Packet Lossy Networks Via Multiple Description Coding

Deep Learning Based Coded Over-the-Air Computation for Personalized Federated Learning

Secure Federated Learning with Model Compression.

Secure Federated Learning over Wireless Communication Networks with Model Compression

FedTC: Enabling Communication-Efficient Federated Learning Via Transform Coding

Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing.

FedSC: Compatible Gradient Compression for Communication-Efficient Federated Learning

Decentralized Federated Learning under Communication Delays.

Federated Learning With Lossy Distributed Source Coding: Analysis and Optimization

Fed-CVLC: Compressing Federated Learning Communications with Variable-Length Codes

Bandwidth-Aware and Overlap-Weighted Compression for Communication-Efficient Federated Learning

FedNC: A Secure and Efficient Federated Learning Method with Network Coding

A Communication-Efficient Federated Learning by Dynamic Quantization and Free-Ride Coding

SVDFed: Enabling Communication-Efficient Federated Learning via Singular-Value-Decomposition.

Adaptive Control of Client Selection and Gradient Compression for Efficient Federated Learning

FedCod: An Efficient Communication Protocol for Cross-Silo Federated Learning with Coding

Federated Learning with Additional Mechanisms on Clients to Reduce Communication Costs

Heterogeneity-Aware Federated Learning with Adaptive Client Selection and Gradient Compression.

M22: A Communication-Efficient Algorithm for Federated Learning Inspired by Rate-Distortion

Content Compression Coding for Federated Learning.

Supplementary File: Cooperative Gradient Coding for Semi-Decentralized Federated Learning