Abstract:Training deep neural networks on large datasets containing high-dimensional data requires a large amount of computation. A solution to this problem is data-parallel distributed training, where a model is replicated into several computational nodes that have access to different chunks of the data. This approach, however, entails high communication rates and latency because of the computed gradients that need to be shared among nodes at every iteration. The problem becomes more pronounced in the case that there is wireless communication between the nodes (i.e. due to the limited network bandwidth). To address this problem, various compression methods have been proposed including sparsification, quantization, and entropy encoding of the gradients. Existing methods leverage the intra-node information redundancy, that is, they compress gradients at each node independently. In contrast, we advocate that the gradients across the nodes are correlated and propose methods to leverage this inter-node redundancy to improve compression efficiency. Depending on the node communication protocol (parameter server or ring-allreduce), we propose two instances of the LGC approach that we coin Learned Gradient Compression (LGC). Our methods exploit an autoencoder (i.e. trained during the first stages of the distributed training) to capture the common information that exists in the gradients of the distributed nodes. We have tested our LGC methods on the image classification and semantic segmentation tasks using different convolutional neural networks (ResNet50, ResNet101, PSPNet) and multiple datasets (ImageNet, Cifar10, CamVid). The ResNet101 model trained for image classification on Cifar10 achieved an accuracy of 93.57%, which is lower than the baseline distributed training with uncompressed gradients only by 0.18%.

Live Gradient Compensation for Evading Stragglers in Distributed Learning

Stochastic Gradient Coding for Straggler Mitigation in Distributed Learning

Gradient Coding in Decentralized Learning for Evading Stragglers

Joint Dynamic Grouping and Gradient Coding for Time-Critical Distributed Machine Learning in Heterogeneous Edge Networks

Distributed Learning based on 1-Bit Gradient Coding in the Presence of Stragglers

Sequential Gradient Coding For Straggler Mitigation

Leveraging partial stragglers within gradient coding

Straggler-aware Distributed Learning: Communication Computation Latency Trade-off

Design and Optimization of Hierarchical Gradient Coding for Distributed Learning at Edge Devices

Fast and Straggler-Tolerant Distributed SGD with Reduced Computation Load

Sparse Gradient Compression For Distributed Sgd

LAG: Lazily Aggregated Gradient for Communication-Efficient Distributed Learning

Intermittent Pulling with Local Compensation for Communication-Efficient Distributed Learning

Heterogeneity-Aware Gradient Coding for Tolerating and Leveraging Stragglers

Communication-Efficient and Byzantine-Robust Distributed Stochastic Learning with Arbitrary Number of Corrupted Workers

Learned Gradient Compression for Distributed Deep Learning

Communication-Efficient Distributed Learning via Sparse and Adaptive Stochastic Gradient

Communication-Efficient Distributed Learning with Local Immediate Error Compensation

Efficient Byzantine-Resilient Stochastic Gradient Desce

Straggler-Resilient Decentralized Learning via Adaptive Asynchronous Updates