Abstract:Distributed machine learning is a mainstream system to learn insights for analytics and intelligence services of many fronts (e.g., health, streaming and business) from their massive operational data. In such a system, multiple workers train over subsets of data and collaboratively derive a global prediction/inference model by iteratively synchronizing their local learning results, e.g., the model gradients, which in turn generates heavy and bursty traffic and results in high communication overhead in cluster networks. Such communication overhead has became the main bottleneck that limits the efficiency of training machine learning models distributedly. In this paper, our key observation is that local gradients learned by workers may have different contributions to global model convergence and executing differential transmission for different gradients can reduce the communication overhead and improve training efficiency. However, existing gradient transmission mechanisms treat all gradients the same, which may lead to long training time. Motivated by our observations, we propose Differential Gradient Transmission (DGT), a contributionaware differential gradient transmission mechanism for efficient distributed learning, which transfers gradients with different transmission quality according to their contributions. In addition to designing a general architecture of DGT, we have proposed a novel algorithm and a novel protocol to facilitate fast model training. Experiments on a cluster with 6 GTX 1080TI GPUs and 1Gbps network show that DGT decreases the model training time by 19.4% on GoogleNet, 34.4% on AlexNet and 36.5% on VGG-11 compared to default gradient transmission on MXNET. Its acceleration is better than the other two related transmission solutions. Besides, DGT works well with different datasets (Fashion-MNIST, Cifar10), different data distributions (IID, non-IID) and different training algorithms (BSP, FedAVG). (C) 2021 Elsevier B.V. All rights reserved.

Mercury: A Simple Transport Layer Scheduler to Accelerate Distributed DNN Training

Accelerating Distributed DNN Training via Transport Layer Scheduling

US-Byte: an Efficient Communication Framework for Scheduling Unequal-Sized Tensor Blocks in Distributed Deep Learning

Mercury: Fast and Optimal Device Placement for Large Deep Learning Models.

A generic communication scheduler for distributed DNN training acceleration

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Mercury: A High-Performance Streaming Graph Method for Broad and Deep Flow Inspection.

Improving the Performance of Distributed MXNet with RDMA.

Prophet: Speeding Up Distributed DNN Training with Predictable Communication Scheduling.

A Multidimensional Communication Scheduling Method for Hybrid Parallel DNN Training

Traffic Management for Distributed Machine Learning in RDMA-enabled Data Center Networks.

Optimizing Deep Learning Frameworks Incrementally to Get Linear Speedup: A Comparison Between IPoIB and RDMA Verbs

Beamer: Stage-Aware Coflow Scheduling to Accelerate Hyper-Parameter Tuning in Deep Learning Clusters

Layer-Wise Partitioning and Merging for Efficient and Scalable Deep Learning

RDMA Based Congestion Control Strategy for DML Training Optimization in Data Center Networks

Effective Scheduler for Distributed DNN Training Based on MapReduce and GPU Cluster

DSANA: A Distributed Machine Learning Acceleration Solution Based on Dynamic Scheduling and Network Acceleration

DGT: A Contribution-Aware Differential Gradient Transmission Mechanism for Distributed Machine Learning

MEIN: A Multicast-Efficient Interconnect Network for Multi-Chiplet DNN Accelerators

Training Acceleration for Deep Neural Networks: A Hybrid Parallelization Strategy

Rationing Bandwidth Resources for Mitigating Network Resource Contention in Distributed DNN Training Clusters.