Abstract:Deep neural networks (DNNs) have gained tremendous attractions as compelling solutions for applications such as image classification, object detection, speech recognition, and so forth. Its great success comes with excessive trainings to make sure the model accuracy is good enough for those applications. Nowadays, it becomes challenging to train a DNN model because of 1) the model size and data size keep increasing, which usually needs more iterations to train; 2) DNN algorithms evolve rapidly, which requires the training phase to be short for a quick deployment. To address those challenges, distributed training platforms have been proposed to leverage massive server nodes for training, with the hope of significant training time reduction. Therefore, scalability is a critical performance metric to evaluate a distributed training platform. Nevertheless, our analysis reveals that traditional server clusters have poor scalability for training due to the traffic congestions within the server and beyond. The intra-server traffic on the I/O fabric can result in severe congestions and skewed quality of service as high performance devices are competing with each other. Moreover, the traffic congestions on the Ethernet for inter-server communication could also incur significant performance degradation. In this work, we devise a novel distributed training platform, EFLOPS, that adopts an algorithm and system co-design methodology to achieve good scalability. A new server architecture is proposed to alleviate the intra-server congestions. Moreover, a new network topology, BiGraph, is proposed to divide the network into two separate parts, so that there is always a direct connection between any nodes from different parts. Finally, accompany with BiGraph, a topology-aware allreduce algorithm is proposed to eliminate the traffic congestion on the direct connection. The experimental results show that eliminating the congestions on network interface can gain up to 11.3xcommunication speedup. The proposed algorithm and topology can provide further improvement up to 6.08x. The overall performance of ResNet-50 training achieves near-linear scalability, and is competitive to the top-rankings of MLPerf results.

ElasticFlow: An elastic serverless training platform for distributed deep learning

ElasticFlow: an Elastic Serverless Training Platform for Distributed Deep Learning.

Design and implementation of efficient distributed deep learning model inference architecture on serverless computation

ElasticDL: A Kubernetes-native Deep Learning Framework with Fault-tolerance and Elastic Scheduling

Elastic Scheduler: Heterogeneous and Dynamic Deep Learning in the Cloud.

Elastic Deep Learning in Multi-Tenant GPU Clusters

EFLOPS: Algorithm and System Co-Design for a High Performance Distributed Training Platform

FedLess: Secure and Scalable Federated Learning Using Serverless Computing

MinFlow: High-performance and Cost-efficient Data Passing for I/O-intensive Stateful Serverless Analytics.

SoCFlow: Efficient and Scalable DNN Training on SoC-Clustered Edge Servers

Effective Elastic Scaling of Deep Learning Workloads

ESG: Pipeline-Conscious Efficient Scheduling of DNN Workflows on Serverless Platforms with Shareable GPUs

Elan: Towards Generic and Efficient Elastic Training for Deep Learning

Elastic-DF: Scaling Performance of DNN Inference in FPGA Clouds through Automatic Partitioning

Λ DNN : Achieving Predictable Distributed DNN Training with Serverless Architectures

Fluid: Dataset Abstraction and Elastic Acceleration for Cloud-native Deep Learning Training Jobs

Large-scale Knowledge Distillation with Elastic Heterogeneous Computing Resources

Energy-Efficient GPU Clusters Scheduling for Deep Learning

DFlow: Efficient Dataflow-based Invocation Workflow Execution for Function-as-a-Service

GreenFlow: A Carbon-Efficient Scheduler for Deep Learning Workloads

NeoFlow: A Flexible Framework for Enabling Efficient Compilation for High Performance DNN Training