Abstract:The speed of deep neural networks training has become a big bottleneck of deep learning research and development. For example, training GoogleNet by ImageNet dataset on one Nvidia K20 GPU needs 21 days. To speed up the training process, the current deep learning systems heavily rely on the hardware accelerators. However, these accelerators have limited on-chip memory compared with CPUs. To handle large datasets, they need to fetch data from either CPU memory or remote processors. We use both self-hosted Intel Knights Landing (KNL) clusters and multi-GPU clusters as our target platforms. From an algorithm aspect, current distributed machine learning systems are mainly designed for cloud systems. These methods are asynchronous because of the slow network and high fault-tolerance requirement on cloud systems. We focus on Elastic Averaging SGD (EASGD) to design algorithms for HPC clusters. Original EASGD used round-robin method for communication and updating. The communication is ordered by the machine rank ID, which is inefficient on HPC clusters. First, we redesign four efficient algorithms for HPC systems to improve EASGD's poor scaling on clusters. Async EASGD, Async MEASGD, and Hogwild EASGD are faster \textcolor{black}{than} their existing counterparts (Async SGD, Async MSGD, and Hogwild SGD, resp.) in all the comparisons. Finally, we design Sync EASGD, which ties for the best performance among all the methods while being deterministic. In addition to the algorithmic improvements, we use some system-algorithm codesign techniques to scale up the algorithms. By reducing the percentage of communication from 87% to 14%, our Sync EASGD achieves 5.3x speedup over original EASGD on the same platform. We get 91.5% weak scaling efficiency on 4253 KNL cores, which is higher than the state-of-the-art implementation.

SAP-SGD: Accelerating Distributed Parallel Training with High Communication Efficiency on Heterogeneous Clusters

WBSP: Addressing Stragglers in Distributed Machine Learning with Worker-Busy Synchronous Parallel

Heter-Train: A Distributed Training Framework Based on Semi-Asynchronous Parallel Mechanism for Heterogeneous Intelligent Transportation Systems

ABS-SGD: A Delayed Synchronous Stochastic Gradient Descent Algorithm with Adaptive Batch Size for Heterogeneous GPU Clusters.

A Hierarchical Communication Algorithm for Distributed Deep Learning Training.

Accelerating Distributed Training in Heterogeneous Clusters via a Straggler-Aware Parameter Server

Scaling Deep Learning on GPU and Knights Landing clusters

XAgg: Accelerating Heterogeneous Distributed Training Through XDP-Based Gradient Aggregation

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Prague: High-Performance Heterogeneity-Aware Asynchronous Decentralized Training

MG-WFBP: Efficient Data Communication for Distributed Synchronous SGD Algorithms.

DaSGD: Squeezing SGD Parallelization Performance in Distributed Training Using Delayed Averaging

Communication-Efficient Distributed Deep Learning with Merged Gradient Sparsification on GPUs

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

Adaptive Elastic Training for Sparse Deep Learning on Heterogeneous Multi-GPU Servers

Accelerating Distributed K-FAC with Smart Parallelism of Computing and Communication Tasks

Hybrid Parameter Update: Alleviating Imbalance Impacts for Distributed Deep Learning.

Robust Fully-Asynchronous Methods for Distributed Training over General Architecture

P4SGD: Programmable Switch Enhanced Model-Parallel Training on Generalized Linear Models on Distributed FPGAs

Layer-wise Adaptive Gradient Sparsification for Distributed Deep Learning with Convergence Guarantees.

FedGSync: Jointly Optimized Weak Synchronization and Gradient Transmission for Fast Distributed Machine Learning in Heterogeneous WAN