Abstract:......................................................................................................................... 3 1 Introduction ............................................................................................................. 3 1.1 Application Background ............................................................................... 4 1.2 Performance Demands for Deep Learning ................................................... 4 1.3 Existing Parallel Frameworks of Deep Learning ......................................... 4 1.4 Chapter Organization ................................................................................... 5 2 Concepts and Categories of Deep Learning ............................................................ 5 2.1 Deep Learning ............................................................................................. 5 2.1.1 Artificial Neural Networks ................................................................ 5 2.1.2 Concept of Deep Learning ................................................................. 7 2.2 Mainstream Deep Learning Models ............................................................. 8 2.2.1 Autoencoders ..................................................................................... 8 2.2.2 Back Propagation ............................................................................... 9 2.2.3 Convolutional Neural Network ........................................................ 11 3 Parallel Optimization for Deep Learning .............................................................. 13 3.1 Convolutional Architecture for Fast Feature Embedding ......................... 13 3.1.1 Introduction ...................................................................................... 13 3.1.2 CUDA Programming ....................................................................... 14 3.1.3 Architecture of Caffe ....................................................................... 17 3.1.4 Parallel Implementation of Convolution in Caffe ............................ 18 3.2 DistBelief .................................................................................................. 20 3.2.1 Introduction of DistBelief ................................................................ 20 3.2.2 Downpour SGD ............................................................................... 20 3.2.4 Sandblaster L-BFGS ........................................................................ 21 3.3 Deep Learning Based-on Multi-GPUs ...................................................... 22 3.3.1 Data Parallelism ............................................................................... 22 3.3.2 Model Parallelism ............................................................................ 23 3.3.3 Data-Model Parallelism ................................................................... 24 3.3.4 Example System of Multi-GPUs ..................................................... 25 4 Discussions ........................................................................................................... 26 4.1 Grand Challenges of Deep Learning with Big Data ................................. 26 4.1.1 Massive Amounts of Training Sample ............................................ 26 4.1.2 Incremental Streaming Data ............................................................ 26 4.1.3 Learning Speed with Big Data ......................................................... 26 4.1.4 Scalability of Deep Models .............................................................. 27 4.2 Future Work .............................................................................................. 27 References .................................................................................................................... 28 Deep Learning and Its Parallelization: Concepts and Instances Xiaqing Li, Guangyan Zhang, Keqin Li, and Weimin Zheng Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

Efficient Distributed Image Recognition Algorithm of Deep Learning Framework TensorFlow

Optimal distributed parallel algorithms for deep learning framework Tensorflow

DISTRIBUTED HIGH-PERFORMANCE COMPUTING METHODS FOR ACCELERATING DEEP LEARNING TRAINING

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Coded Parallelism for Distributed Deep Learning.

Communication-Efficient Distributed Deep Learning: A Comprehensive Survey

SingleCaffe: an Efficient Framework for Deep Learning on a Single Node

Deep Learning and Its Parallelization

Communication Optimization for Distributed Training: Architecture, Advances, and Opportunities

EFLOPS: Algorithm and System Co-Design for a High Performance Distributed Training Platform

TensorOpt: Exploring the Tradeoffs in Distributed DNN Training With Auto-Parallelism

TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems

Communication-Efficient Large-Scale Distributed Deep Learning: A Comprehensive Survey

OSDP: Optimal Sharded Data Parallel for Distributed Deep Learning

On Efficient Training of Large-Scale Deep Learning Models: A Literature Review

TensorFlow: A system for large-scale machine learning

Parallel Blockwise Knowledge Distillation for Deep Neural Network Compression

Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks

Research on Optimization of Image Recognition Algorithm Based on Deep Learning

Communication Patterns in Distributed Deep Learning

PowerAI DDL