Abstract:Top-K sparsification-based compression techniques are popular and powerful for reducing communication costs in federated learning (FL). However, existing Top-K sparsification-based compression methods suffer from two critical issues that severely hinder their implementation, particularly in the context of FL, which often involves a vast number of resource-constrained devices: 1) the low compressibility of the Top-K parameter’s indexes significantly limits the overall compression ratio (CR) and 2) the residual accumulation techniques used to maintain the model quality consume huge memory resources. To address these issues, we propose a novel FL compression framework, named FedComp, for deep neural networks (DNNs). FedComp achieves a higher communication CR while maintaining comparable model quality at low memory cost. Specifically, FedComp incorporates the following three key components: 1) a tensor-wise index-sharing mechanism that greatly reduces the index proportion by sharing one index among multiple elements of the tensor; 2) a fine-grained parameters packing strategy that reduces the transmission of duplicate value and index by considering their properties, thereby further reducing the overall communication cost; and 3) a residual compressor that significantly reduces memory cost by enhancing the compressibility of floating-point residuals and achieving a high CR with a lossless encoding scheme. Experiments on mainstream machine learning (ML) tasks with different DNN structures and datasets demonstrate that our proposed FedComp outperforms the state-of-the-art FL compression algorithms by achieving a higher communication CR of up to $28.5\times $ while reducing memory costs by $21.04\times $ – $50.59\times $ on the local residual model, without degrading FL training performance.

A Deep Gradient Compression Model Based on Tensor-Train Decomposition for Federated Learning

Understanding the Training Dynamics in Federated Deep Learning via Aggregation Weight Optimization

Secure Federated Learning with Model Compression.

Secure Federated Learning over Wireless Communication Networks with Model Compression

FedDGP: Disentangling Global and Personal Models for Federated Learning

Deep Learning Based Coded Over-the-Air Computation for Personalized Federated Learning

Wyner-Ziv Gradient Compression for Federated Learning

Efficient Model Compression for Hierarchical Federated Learning

CG-FedLLM: How to Compress Gradients in Federated Fune-tuning for Large Language Models

Efficient Wireless Federated Learning via Low-Rank Gradient Factorization

Stochastic gradient compression for federated learning over wireless network

Data-Aware Gradient Compression for FL in Communication-Constrained Mobile Computing

Deep Learning Model Compression with Rank Reduction in Tensor Decomposition.

Ternary Compression for Communication-Efficient Federated Learning.

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training.

Communication-Efficient Federated Learning through Adaptive Weight Clustering and Server-Side Distillation

Toward Efficient Federated Learning in Multi-Channeled Mobile Edge Network with Layerd Gradient Compression

Content Compression Coding for Federated Learning.

Federated Learning With Lossy Distributed Source Coding: Analysis and Optimization

FedComp: A Federated Learning Compression Framework for Resource-Constrained Edge Computing Devices

Efficient Model Compression via Global Sparsification for Over-the-Air Federated Learning.