Abstract:In recent years, deep neural networks (DNN) have demonstrated significant business impact in large scale analysis and classification tasks such as speech recognition, visual object detection, pattern extraction, etc. Training of large DNNs, however, is universally considered as time consuming and computationally intensive task that demands datacenter-scale computational resources recruited for many days. Here we propose a concept of resistive processing unit (RPU) devices that can potentially accelerate DNN training by orders of magnitude while using much less power. The proposed RPU device can store and update the weight values locally thus minimizing data movement during training and allowing to fully exploit the locality and the parallelism of the training algorithm. We identify the RPU device and system specifications for implementation of an accelerator chip for DNN training in a realistic CMOS-compatible technology. For large DNNs with about 1 billion weights this massively parallel RPU architecture can achieve acceleration factors of 30,000X compared to state-of-the-art microprocessors while providing power efficiency of 84,000 GigaOps/s/W. Problems that currently require days of training on a datacenter-size cluster with thousands of machines can be addressed within hours on a single RPU accelerator. A system consisted of a cluster of RPU accelerators will be able to tackle Big Data problems with trillions of parameters that is impossible to address today like, for example, natural speech recognition and translation between all world languages, real-time analytics on large streams of business and scientific data, integration and analysis of multimodal sensory data flows from massive number of IoT (Internet of Things) sensors.

ACCELERATING THE TRAINING OF ARTIFICIAL NEURAL NETWORKS USING DATA PARALLELIZATION

AccEPT: an Acceleration Scheme for Speeding Up Edge Pipeline-parallel Training

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

Parallax: Sparsity-aware Data Parallel Training of Deep Neural Networks

Using GPI-2 for Distributed Memory Paralleliziation of the Caffe Toolbox to Speed up Deep Neural Network Training

Accelerating neural network training with distributed asynchronous and selective optimization (DASO)

Cyclic Data Parallelism for Efficient Parallelism of Deep Neural Networks

Efficient and Robust Parallel DNN Training through Model Parallelism on Multi-GPU Platform

A Hybrid Parallelization Approach for Distributed and Scalable Deep Learning

HierTrain: Fast Hierarchical Edge AI Learning with Hybrid Parallelism in Mobile-Edge-Cloud Computing

GLP4NN: A Convergence-invariant and Network-agnostic Light-weight Parallelization Framework for Deep Neural Networks on Modern GPUs

Reducing Data Motion to Accelerate the Training of Deep Neural Networks

Accelerating DNN Training in Wireless Federated Edge Learning Systems

Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices

EdgeSP: Scalable Multi-device Parallel DNN Inference on Heterogeneous Edge Clusters

DAHA: Accelerating GNN Training with Data and Hardware Aware Execution Planning

Brief Announcement: On the Limits of Parallelizing Convolutional Neural Networks on GPUs

A Practical Implementation of GPU based Accelerator for Deep Neural Networks

Near-Linear Scaling Data Parallel Training with Overlapping-Aware Gradient Compression

HPH: Hybrid Parallelism on Heterogeneous Clusters for Accelerating Large-scale DNNs Training.

TAP: Accelerating Large-Scale DNN Training Through Tensor Automatic Parallelisation