Abstract: The ever-growing model size and scale of compute have attracted increasing interests in training deep learning models over multiple nodes. However, when it comes to training on cloud clusters, especially across remote clusters, huge challenges are faced. In this work, we introduce a general framework, Nebula-I, for collaboratively training deep learning models over remote heterogeneous clusters, the connections between which are low-bandwidth wide area networks (WANs). We took natural language processing (NLP) as an example to show how Nebula-I works in different training phases that include: a) pre-training a multilingual language model using two remote clusters; and b) fine-tuning a machine translation model using knowledge distilled from pre-trained models, which run through the most popular paradigm of recent deep learning. To balance the accuracy and communication efficiency, in Nebula-I, parameter-efficient training strategies, hybrid parallel computing methods and adaptive communication acceleration techniques are jointly applied. Meanwhile, security strategies are employed to guarantee the safety, reliability and privacy in intra-cluster computation and inter-cluster communication. Nebula-I is implemented with the PaddlePaddle deep learning framework, which can support collaborative training over heterogeneous hardware, e.g. GPU and NPU. Experiments demonstrate that the proposed framework could substantially maximize the training efficiency while preserving satisfactory NLP performance. By using Nebula-I, users can run large-scale training tasks over cloud clusters with minimum developments, and the utility of existed large pre-trained models could be further promoted. We also introduced new state-of-the-art results on cross-lingual natural language inference tasks, which are generated based upon a novel learning framework and Nebula-I.

Rationing Bandwidth Resources for Mitigating Network Resource Contention in Distributed DNN Training Clusters.

Nebula-I: A General Framework for Collaboratively Training Deep Learning Models on Low-Bandwidth Cloud Clusters

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Themis: A Network Bandwidth-Aware Collective Scheduling Policy for Distributed Training of DL Models

Improving Cluster Utilization Through Adaptive Resource Management for Deep Neural Network and CPU Jobs Colocation

Performance Modeling and Analysis of Distributed Deep Neural Network Training with Parameter Server

spotDNN: Provisioning Spot Instances for Predictable Distributed DNN Training in the Cloud.

Traffic Management for Distributed Machine Learning in RDMA-enabled Data Center Networks.

A Sum-of-Ratios Multi-Dimensional-Knapsack Decomposition for DNN Resource Scheduling

A Novel Co-design Peta-scale Heterogeneous Cluster for Deep Learning Training

Libra: Contention-Aware GPU Thread Allocation for Data Parallel Training in High Speed Networks.

Joint Dynamic Data and Model Parallelism for Distributed Training of DNNs over Heterogeneous Infrastructure

Isolated Scheduling for Distributed Training Tasks in GPU Clusters

A Unified Architecture for Accelerating Distributed DNN Training in Heterogeneous GPU/CPU Clusters

Optimizing Network Performance for Distributed DNN Training on GPU Clusters: ImageNet/AlexNet Training in 1.5 Minutes

DLB: A Dynamic Load Balance Strategy for Distributed Training of Deep Neural Networks

Benchmarking Resource Usage for Efficient Distributed Deep Learning

MLTCP: Congestion Control for DNN Training

CODA: Improving Resource Utilization by Slimming and Co-locating DNN and CPU Jobs

DDPQN: An Efficient DNN Offloading Strategy in Local-Edge-Cloud Collaborative Environments

DBS: Dynamic Batch Size For Distributed Deep Neural Network Training