Abstract:With the increment of data sets and models sizes, distributed deep learning has been proposed to accelerate training and improve the accuracy of DNN models. The parameter server framework is a popular collaborative architecture for data-parallel training, which works well for homogeneous environments by properly aggregating the computation/communication capabilities of different workers. However, in heterogeneous environments, the resources of different workers vary a lot. Some stragglers may seriously limit the whole speed, which impacts the overall training process. In this paper, we propose an adaptive multi-stage distributed deep learning training framework, named ASHL, for heterogeneous environments. First, a profiling scheme is proposed to capture the capabilities of each worker to reasonably plan the training and communication tasks on each worker, and lay the foundation for the formal training. Second, a hybrid-mode training scheme (i.e., coarse-grained and fined-grained training) is proposed to balance the model accuracy and training speed. The coarse-grained training scheme (named AHL) adopts an asynchronous communication strategy, which involves less frequent communications. Its main goal is to make the model quickly converge to a certain level. The fine-grained training stage (named SHL) uses a semi-asynchronous communication strategy and adopts a high communication frequency. Its main goal is to improve the model convergence effect. Finally, a compression-based communication scheme is proposed to further increase the communication efficiency of the training process. Our experimental results show that ASHL reduces the overall training time by more than 35% to converge to the same degree and has better generalization ability compared with state-of-the-art schemes like ADSP.

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Heter-Train: A Distributed Training Framework Based on Semi-Asynchronous Parallel Mechanism for Heterogeneous Intelligent Transportation Systems

ASHL: an Adaptive Multi-Stage Distributed Deep Learning Training Scheme for Heterogeneous Environments

Optimizing execution for pipelined‐based distributed deep learning in a heterogeneously networked GPU cluster

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

Coded Parallelism for Distributed Deep Learning.

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Accelerating Training For Distributed Deep Neural Networks In Mapreduce

HeterPS: Distributed deep learning with reinforcement learning based scheduling in heterogeneous environments

EP4DDL: addressing straggler problem in heterogeneous distributed deep learning

MP-DPS: Adaptive Distributed Training for Deep Learning Based on Node Merging and Path Prediction

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

Model-Aware Parallelization Strategy for Deep Neural Networks' Distributed Training

Scheduling-Efficient Framework for Neural Network on Heterogeneous Distributed Systems and Mobile Edge Computing Systems

A Load-Balancing Strategy Based on Multi-Task Learning in a Distributed Training Environment

PipePar: A Pipelined Hybrid Parallel Approach for Accelerating Distributed DNN Training

Modeling the Training Iteration Time for Heterogeneous Distributed Deep Learning Systems.

HAP: SPMD DNN Training on Heterogeneous GPU Clusters with Automated Program Synthesis

DSANA: A Distributed Machine Learning Acceleration Solution Based on Dynamic Scheduling and Network Acceleration

Effective Scheduler for Distributed DNN Training Based on MapReduce and GPU Cluster