Abstract:The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference -- i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark for DNN training, called TBD (TBD is short for Training Benchmark for DNNs), that uses a representative set of DNN models that cover a wide range of machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) by performing an extensive performance analysis of training these different applications on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine). TBD currently covers six major application domains and eight different state-of-the-art models. We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of new and existing metrics and methodologies to analyze the results, and utilization of domain specific characteristics of DNN training. We also build a new set of tools for memory profiling in all three major frameworks; much needed tools that can finally shed some light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. By using our tools and methodologies, we make several important observations and recommendations on where the future research and optimization of DNN training should be focused.

A domain-specific supercomputer for training deep neural networks

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

DaDianNao: A Machine-Learning Supercomputer

Benchmarking TPU, GPU, and CPU Platforms for Deep Learning

Deep Learning on Edge TPUs

Exploring Deep Neural Networks on Edge TPU

Training EfficientNets at Supercomputer Scale: 83% ImageNet Top-1 Accuracy in One Hour

SuperNeurons: Dynamic GPU Memory Management for Training Deep Neural Networks

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

High-resolution imaging on TPUs

Harnessing Manycore Processors with Distributed Memory for Accelerated Training of Sparse and Recurrent Models

Implementation and evaluation of deep neural networks (DNN) on mainstream heterogeneous systems

Exploration of TPUs for AI Applications

A Carbon-Nanotube-based Tensor Processing Unit

Acceleration of Deep Neural Network Training with Resistive Cross-Point Devices

TBD: Benchmarking and Analyzing Deep Neural Network Training

GPTPU: Accelerating Applications using Edge Tensor Processing Units

A High Energy-Efficiency Multi-core Neuromorphic Architecture for Deep SNN Training

Parallel Training of Pre-Trained Models Via Chunk-Based Dynamic Memory Management

A 28nm 276.55TFLOPS/W Sparse Deep-Neural-Network Training Processor with Implicit Redundancy Speculation and Batch Normalization Reformulation

CATERPILLAR: Coarse Grain Reconfigurable Architecture for Accelerating the Training of Deep Neural Networks