Abstract: Deep learning (DL) training-as-a-service (TaaS) is an important emerging industrial workload. The unique challenge of TaaS is that it must satisfy a wide range of customers who have no experience and resources to tune DL hyper-parameters, and meticulous tuning for each user's dataset is prohibitively expensive. Therefore, TaaS hyper-parameters must be fixed with values that are applicable to all users. IBM Watson Natural Language Classifier (NLC) service, the most popular IBM cognitive service used by thousands of enterprise-level clients around the globe, is a typical TaaS service. By evaluating the NLC workloads, we show that only the conservative hyper-parameter setup (e.g., small mini-batch size and small learning rate) can guarantee acceptable model accuracy for a wide range of customers. We further justify theoretically why such a setup guarantees better model convergence in general. Unfortunately, the small mini-batch size causes a high volume of communication traffic in a parameter-server based system. We characterize the high communication bandwidth requirement of TaaS using representative industrial deep learning workloads and demonstrate that none of the state-of-the-art scale-up or scale-out solutions can satisfy such a requirement. We then present GaDei, an optimized shared-memory based scale-up parameter server design. We prove that the designed protocol is deadlock-free and it processes each gradient exactly once. Our implementation is evaluated on both commercial benchmarks and public benchmarks to demonstrate that it significantly outperforms the state-of-the-art parameter-server based implementation while maintaining the required accuracy and our implementation reaches near the best possible runtime performance, constrained only by the hardware limitation. Furthermore, to the best of our knowledge, GaDei is the only scale-up DL system that provides fault-tolerance.

BigDL 2.0: Seamless Scaling of AI Pipelines from Laptops to Distributed Cluster

BigDL: A Distributed Deep Learning Framework for Big Data

XDL: an industrial deep learning framework for high-dimensional sparse data

PowerAI DDL

Accelerating Large Scale Deep Learning Inference Through DeepCPU at Microsoft.

Deep Learning At Scale and At Ease

AutoDDL: Automatic Distributed Deep Learning With Near-Optimal Bandwidth Cost

ElasticDL: A Kubernetes-native Deep Learning Framework with Fault-tolerance and Elastic Scheduling

Bigflow: A General Optimization Layer for Distributed Computing Frameworks

WidePipe: High-Throughput Deep Learning Inference System on a Cluster of Neural Processing Units

Accelerating Deep Learning Systems Via Critical Set Identification and Model Compression.

GaDei: On Scale-up Training As A Service For Deep Learning

Accelerating End-to-End Deep Learning Workflow With Codesign of Data Preprocessing and Scheduling.

Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Training

High Performance I/O For Large Scale Deep Learning

Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning

Fire-Flyer AI-HPC: A Cost-Effective Software-Hardware Co-Design for Deep Learning

HPDL: Towards a General Framework for High-performance Distributed Deep Learning.

Design and implementation of DeepDSL: A DSL for deep learning

Deep Learning on Operational Facility Data Related to Large-Scale Distributed Area Scientific Workflows