Abstract:Federated Learning (FL) communication costs hinge on communication frequency, device count, and per-communication-round costs. Ideally, minimizing these within the device cluster tolerance significantly curtails data traffic. Among various methods to reduce per-communication-round costs, gradient compression stands out. Gradient compression concentrates on the changes in the model rather than the model parameters. This not only prevents synchronization issues caused by varying compression levels across devices, but also incurs only minor precision loss. This makes it especially apt for distributed scenarios like FL. Existing gradient compression methods are tailored to capitalize on elements nearing zero in high-frequency settings, aiming for sparse representation and efficient encoding. Yet, in low-frequency scenarios where elements deviate from zero, this strong emphasis on sparsity results in significant compression errors. To resolve this, we propose Federated Statistical Compression (FedSC) shifts the focus from sparsity. Designed specifically for low-frequency settings, it hones in on the inherent statistical characteristics and relationships among the gradient elements. It extracts statistical information from each gradient, replacing the original gradient sent to the central server. We further introduce hierarchical estimation to improve accuracy, grouping elements in a gradient by layer label. Experimentally, FedSC outperforms most existing gradient compression algorithms for FL, converging with fewer communication bits while maintaining high model accuracy.

Enabling Communication-Efficient Federated Learning via Distributed Compressed Sensing.

Communication-Efficient and Secure Federated Learning Based on Adaptive One-Bit Compressed Sensing

FedSC: Compatible Gradient Compression for Communication-Efficient Federated Learning

Secure Federated Learning with Model Compression.

Secure Federated Learning over Wireless Communication Networks with Model Compression

SVDFed: Enabling Communication-Efficient Federated Learning via Singular-Value-Decomposition.

Communication-Efficient Federated Learning via Quantized Compressed Sensing

FedTC: Enabling Communication-Efficient Federated Learning Via Transform Coding

Stochastic gradient compression for federated learning over wireless network

Communication-Efficient Federated Learning Based on Chained Secure Multiparty Computing and Compressed Sensing.

FedMDC: Enabling Communication-Efficient Federated Learning over Packet Lossy Networks Via Multiple Description Coding

Communication-efficient federated learning based on compressed sensing and ternary quantization

Personalized Federated Learning with Communication Compression

A Novel Adaptive Gradient Compression Approach for Communication-Efficient Federated Learning

Communication-Efficient Federated Learning with Dual-Side Low-Rank Compression

Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computing

Decentralized Federated Learning: Balancing Communication and Computing Costs

1-Bit Compressive Sensing for Efficient Federated Learning Over the Air

Communication-efficient Federated Learning with Single-Step Synthetic Features Compressor for Faster Convergence

Efficient Wireless Federated Learning via Low-Rank Gradient Factorization

Communication-efficient Federated Learning Through 1-Bit Compressive Sensing and Analog Aggregation.