Abstract:Training deep neural networks on large datasets containing high-dimensional data requires a large amount of computation. A solution to this problem is data-parallel distributed training, where a model is replicated into several computational nodes that have access to different chunks of the data. This approach, however, entails high communication rates and latency because of the computed gradients that need to be shared among nodes at every iteration. The problem becomes more pronounced in the case that there is wireless communication between the nodes (i.e. due to the limited network bandwidth). To address this problem, various compression methods have been proposed including sparsification, quantization, and entropy encoding of the gradients. Existing methods leverage the intra-node information redundancy, that is, they compress gradients at each node independently. In contrast, we advocate that the gradients across the nodes are correlated and propose methods to leverage this inter-node redundancy to improve compression efficiency. Depending on the node communication protocol (parameter server or ring-allreduce), we propose two instances of the LGC approach that we coin Learned Gradient Compression (LGC). Our methods exploit an autoencoder (i.e. trained during the first stages of the distributed training) to capture the common information that exists in the gradients of the distributed nodes. We have tested our LGC methods on the image classification and semantic segmentation tasks using different convolutional neural networks (ResNet50, ResNet101, PSPNet) and multiple datasets (ImageNet, Cifar10, CamVid). The ResNet101 model trained for image classification on Cifar10 achieved an accuracy of 93.57%, which is lower than the baseline distributed training with uncompressed gradients only by 0.18%.

A Novel Adaptive Gradient Compression Scheme: Reducing the Communication Overhead for Distributed Deep Learning in the Internet of Things

Efficient Partitioning and Communication Scheme-Based Distributed Edge Computing to Accelerate Deep Neural Network

A Partition Based Gradient Compression Algorithm for Distributed Training in AIoT

Compressed Communication for Distributed Training: Adaptive Methods and System

AC-SGD: Adaptively Compressed SGD for Communication-Efficient Distributed Learning

Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Training.

Understanding Sensor Data Using Deep Learning Methods on Resource-Constrained Edge Devices.

An Efficient Bandwidth-Adaptive Gradient Compression Algorithm for Distributed Training of Deep Neural Networks

Near-Lossless Gradient Compression for Data-Parallel Distributed DNN Training

Evaluation and Optimization of Gradient Compression for Distributed Deep Learning

On-Demand Deep Model Compression for Mobile Devices

Flexible Communication for Optimal Distributed Learning over Unpredictable Networks

A Novel Adaptive Gradient Compression Approach for Communication-Efficient Federated Learning

Adaptive Compression for Communication-Efficient Distributed Training

Adaptive Batchsize Selection and Gradient Compression for Wireless Federated Learning

Learned Gradient Compression for Distributed Deep Learning

DAGC: Data-Aware Adaptive Gradient Compression.

Stochastic gradient compression for federated learning over wireless network

DAGC: Data-Volume-Aware Adaptive Sparsification Gradient Compression for Distributed Machine Learning in Mobile Computing

Gradient Compression Supercharged High-Performance Data Parallel DNN Training.

Accordion: Adaptive Gradient Communication via Critical Learning Regime Identification