Abstract:The recent popularity of deep neural networks (DNNs) has generated a lot of research interest in performing DNN-related computation efficiently. However, the primary focus is usually very narrow and limited to (i) inference -- i.e. how to efficiently execute already trained models and (ii) image classification networks as the primary benchmark for evaluation. Our primary goal in this work is to break this myopic view by (i) proposing a new benchmark for DNN training, called TBD (TBD is short for Training Benchmark for DNNs), that uses a representative set of DNN models that cover a wide range of machine learning applications: image classification, machine translation, speech recognition, object detection, adversarial networks, reinforcement learning, and (ii) by performing an extensive performance analysis of training these different applications on three major deep learning frameworks (TensorFlow, MXNet, CNTK) across different hardware configurations (single-GPU, multi-GPU, and multi-machine). TBD currently covers six major application domains and eight different state-of-the-art models. We present a new toolchain for performance analysis for these models that combines the targeted usage of existing performance analysis tools, careful selection of new and existing metrics and methodologies to analyze the results, and utilization of domain specific characteristics of DNN training. We also build a new set of tools for memory profiling in all three major frameworks; much needed tools that can finally shed some light on precisely how much memory is consumed by different data structures (weights, activations, gradients, workspace) in DNN training. By using our tools and methodologies, we make several important observations and recommendations on where the future research and optimization of DNN training should be focused.

DVTest: Deep Neural Network Visualization Testing Framework

NeuralVis: Visualizing and Interpreting Deep Learning Models

DeepIA: An Interpretability Analysis based Test Data Generation Method for DNN

DeepMutation: Mutation Testing of Deep Learning Systems

An In-Situ Visual Analytics Framework for Deep Neural Networks

A walk in the black-box: 3D visualization of large neural networks in virtual reality

Dissecting Deep Learning Networks—Visualizing Mutual Information

Understanding Neural Networks Through Deep Visualization

Test4Deep: an Effective White-Box Testing for Deep Neural Networks

A Search-Based Testing Framework for Deep Neural Networks of Source Code Embedding

DeepGD: A Multi-Objective Black-Box Test Selection Approach for Deep Neural Networks

DevMuT: Testing Deep Learning Framework Via Developer Expertise-Based Mutation

Unveiling Hidden DNN Defects with Decision-Based Metamorphic Testing

Testing Deep Learning Models: A First Comparative Study of Multiple Testing Techniques

Audee: Automated Testing for Deep Learning Frameworks

MuNN: Mutation Analysis of Neural Networks

DiG-IN: Diffusion Guidance for Investigating Networks -- Uncovering Classifier Differences Neuron Visualisations and Visual Counterfactual Explanations

DeepViT: Towards Deeper Vision Transformer

TBD: Benchmarking and Analyzing Deep Neural Network Training

DeepGini: prioritizing massive tests to enhance the robustness of deep neural networks