Abstract:......................................................................................................................... 3 1 Introduction ............................................................................................................. 3 1.1 Application Background ............................................................................... 4 1.2 Performance Demands for Deep Learning ................................................... 4 1.3 Existing Parallel Frameworks of Deep Learning ......................................... 4 1.4 Chapter Organization ................................................................................... 5 2 Concepts and Categories of Deep Learning ............................................................ 5 2.1 Deep Learning ............................................................................................. 5 2.1.1 Artificial Neural Networks ................................................................ 5 2.1.2 Concept of Deep Learning ................................................................. 7 2.2 Mainstream Deep Learning Models ............................................................. 8 2.2.1 Autoencoders ..................................................................................... 8 2.2.2 Back Propagation ............................................................................... 9 2.2.3 Convolutional Neural Network ........................................................ 11 3 Parallel Optimization for Deep Learning .............................................................. 13 3.1 Convolutional Architecture for Fast Feature Embedding ......................... 13 3.1.1 Introduction ...................................................................................... 13 3.1.2 CUDA Programming ....................................................................... 14 3.1.3 Architecture of Caffe ....................................................................... 17 3.1.4 Parallel Implementation of Convolution in Caffe ............................ 18 3.2 DistBelief .................................................................................................. 20 3.2.1 Introduction of DistBelief ................................................................ 20 3.2.2 Downpour SGD ............................................................................... 20 3.2.4 Sandblaster L-BFGS ........................................................................ 21 3.3 Deep Learning Based-on Multi-GPUs ...................................................... 22 3.3.1 Data Parallelism ............................................................................... 22 3.3.2 Model Parallelism ............................................................................ 23 3.3.3 Data-Model Parallelism ................................................................... 24 3.3.4 Example System of Multi-GPUs ..................................................... 25 4 Discussions ........................................................................................................... 26 4.1 Grand Challenges of Deep Learning with Big Data ................................. 26 4.1.1 Massive Amounts of Training Sample ............................................ 26 4.1.2 Incremental Streaming Data ............................................................ 26 4.1.3 Learning Speed with Big Data ......................................................... 26 4.1.4 Scalability of Deep Models .............................................................. 27 4.2 Future Work .............................................................................................. 27 References .................................................................................................................... 28 Deep Learning and Its Parallelization: Concepts and Instances Xiaqing Li, Guangyan Zhang, Keqin Li, and Weimin Zheng Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

Dragon: A Computation Graph Virtual Machine Based Deep Learning Framework

Dragon-Alpha&cu32: A Java-based Tensor Computing Framework With its High-Performance CUDA Library

Deep Learning with Dynamic Computation Graphs

Cavs: A Vertex-centric Programming Interface for Dynamic Neural Networks.

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

A Data-Centric Optimization Framework for Machine Learning

Coded Parallelism for Distributed Deep Learning.

An Orchestrated Empirical Study on Deep Learning Frameworks and Platforms

DeepDSL: A Compilation-based Domain-Specific Language for Deep Learning

A Multi-Level Framework for Accelerating Training Transformer Models

Cavs: An Efficient Runtime System For Dynamic Neural Networks

SingleCaffe: an Efficient Framework for Deep Learning on a Single Node

Deep Learning and Its Parallelization

Computation-efficient Deep Learning for Computer Vision: A Survey

Performance Modeling and Evaluation of Distributed Deep Learning Frameworks on GPUs

Collage: Seamless Integration of Deep Learning Backends with Automatic Placement

Efficient Deep Learning Infrastructures for Embedded Computing Systems: A Comprehensive Survey and Future Envision

Automated Deep Learning Optimization via DSL-Based Source Code Transformation

NeuGraph: Parallel Deep Neural Network Computation on Large Graphs

Model Compression and Acceleration for Deep Neural Networks: the Principles, Progress, and Challenges

An In-depth Comparison of Compilers for Deep Neural Networks on Hardware