Abstract:......................................................................................................................... 3 1 Introduction ............................................................................................................. 3 1.1 Application Background ............................................................................... 4 1.2 Performance Demands for Deep Learning ................................................... 4 1.3 Existing Parallel Frameworks of Deep Learning ......................................... 4 1.4 Chapter Organization ................................................................................... 5 2 Concepts and Categories of Deep Learning ............................................................ 5 2.1 Deep Learning ............................................................................................. 5 2.1.1 Artificial Neural Networks ................................................................ 5 2.1.2 Concept of Deep Learning ................................................................. 7 2.2 Mainstream Deep Learning Models ............................................................. 8 2.2.1 Autoencoders ..................................................................................... 8 2.2.2 Back Propagation ............................................................................... 9 2.2.3 Convolutional Neural Network ........................................................ 11 3 Parallel Optimization for Deep Learning .............................................................. 13 3.1 Convolutional Architecture for Fast Feature Embedding ......................... 13 3.1.1 Introduction ...................................................................................... 13 3.1.2 CUDA Programming ....................................................................... 14 3.1.3 Architecture of Caffe ....................................................................... 17 3.1.4 Parallel Implementation of Convolution in Caffe ............................ 18 3.2 DistBelief .................................................................................................. 20 3.2.1 Introduction of DistBelief ................................................................ 20 3.2.2 Downpour SGD ............................................................................... 20 3.2.4 Sandblaster L-BFGS ........................................................................ 21 3.3 Deep Learning Based-on Multi-GPUs ...................................................... 22 3.3.1 Data Parallelism ............................................................................... 22 3.3.2 Model Parallelism ............................................................................ 23 3.3.3 Data-Model Parallelism ................................................................... 24 3.3.4 Example System of Multi-GPUs ..................................................... 25 4 Discussions ........................................................................................................... 26 4.1 Grand Challenges of Deep Learning with Big Data ................................. 26 4.1.1 Massive Amounts of Training Sample ............................................ 26 4.1.2 Incremental Streaming Data ............................................................ 26 4.1.3 Learning Speed with Big Data ......................................................... 26 4.1.4 Scalability of Deep Models .............................................................. 27 4.2 Future Work .............................................................................................. 27 References .................................................................................................................... 28 Deep Learning and Its Parallelization: Concepts and Instances Xiaqing Li, Guangyan Zhang, Keqin Li, and Weimin Zheng Department of Computer Science and Technology, Tsinghua University, Beijing 100084, China

Parallel Learning: Overview and Perspective for Computational Learning Across Syn2Real and Sim2Real

Parallel Learning: a Perspective and a Framework

From Virtuality To Reality: A Learning-based Point Cloud Labeling Method With Synthesis Scene

Parallel Learning - A New Framework for Machine Learning

Parallel Vision ⊇ Image Synthesis/Augmentation

A novel simulation reality closed loop learning framework for autonomous robot skill learning

How Simulation Helps Autonomous Driving:A Survey of Sim2real, Digital Twins, and Parallel Intelligence

Parallel Driving with Big Models and Foundation Intelligence in Cyber-Physical-Social Spaces

Parallel Vision for Perception and Understanding of Complex Scenes: Methods, Framework, and Perspectives

Multi-view Learning Overview: Recent Progress and New Challenges

Sim-to-Real Transfer in Deep Reinforcement Learning for Robotics: a Survey

Parallel Point Clouds: Hybrid Point Cloud Generation and 3D Model Enhancement via Virtual–Real Integration

Deep Learning and Its Parallelization

Parallel learner: A practical deep reinforcement learning framework for multi-scenario games

A Survey on Multi-view Learning

Training and Testing Object Detectors with Virtual Images

Parallel reinforcement learning: a framework and case study

Natural Language Can Help Bridge the Sim2Real Gap

Overview of deep learning application on visual SLAM

Learning Interactive Real-World Simulators

Close the Sim2real Gap via Physically-based Structured Light Synthetic Data Simulation