Abstract:Edge devices are seeing tremendous growth in sensing and computational capabilities. Running state-of-the-art deep neural network (NN) based data processing on multi-core CPU processors, embedded Graphics Processing Units (GPU), Tensor Processing Units (TPU), Neural Processing Units (NPU), Deep Learning Accelerators (DLA) etc., edge devices are now able to handle heavy data computations with limited or without cloud connectivity. In addition to hardware resources, software frameworks that optimize a trained neural network (NN) model through weight clustering and pruning, weight and input-output quantization to fewer bits, fusing NN layers etc., for more efficient execution of NN inferences on edge platforms, play an important role in making machine learning at the edge (namely EdgeML) a reality. This paper is a first effort in characterizing these software frameworks for DNN inference optimizations on edge devices, especially edge GPUs which are now ubiquitously used in all embedded deep learning systems. The interactions between software optimizations and the underlying GPU hardware is carefully examined. As most NN optimization engines are proprietary softwares with undocumented internal details in the public domain, our empirical analysis on real embedded GPU platforms using a variety of widely used DNNs, provide various interesting findings. We observe tremendous performance gain and non-negligible accuracy gain from the software optimizations, but also find highly unexpected non-deterministic behaviors such as different outputs on same inputs or increased execution latency for same NN model on more powerful hardware platforms. Application developers using these proprietary software optimization engines, would benefit from our analysis and the discussed implications of our findings, with examples from real applications like intelligent traffic intersection control and Advanced Driving Assistance Systems (ADAS). There are important implications of our findings on performance modeling and prediction research too, that focus on micro-architecture modeling based application performance prediction, but should now additionally consider optimization engines that this paper examines.

Tensor Processing Unit Reliability Dependence on Temperature and Radiation Source

Two Watts is All You Need: Enabling In-Detector Real-Time Machine Learning for Neutrino Telescopes Via Edge Computing

Exploration of TPUs for AI Applications

Exploring Hardware Fault Impacts on Different Real Number Representations of the Structural Resilience of TCUs in GPUs

Evaluation of Xilinx Deep Learning Processing Unit under Neutron Irradiation

Demystifying TensorRT: Characterizing Neural Network Inference Engine on Nvidia Edge Devices

Fast Object Detection with a Machine Learning Edge Device

GPTPU: Accelerating Applications using Edge Tensor Processing Units

Characterization of Single-Event Effects in a Microcontroller with an Artificial Neural Network Accelerator

Deep Learning on Edge TPUs

Exploring Deep Neural Networks on Edge TPU

A Carbon-Nanotube-based Tensor Processing Unit

High-resolution imaging on TPUs

Heterogeneous Integration of In-Memory Analog Computing Architectures with Tensor Processing Units

Evaluation of NVIDIA Xavier NX Platform for Real-Time Image Processing for Plasma Diagnostics

Single Event Effects Assessment of UltraScale+ MPSoC Systems under Atmospheric Radiation

TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddings

Deep Learning for Real-Time Chip Temperature and Power Predictions

DOCTOR: Dynamic On-Chip Temporal Variation Remediation Toward Self-Corrected Photonic Tensor Accelerators

Benchmarking Edge Computing Devices for Grape Bunches and Trunks Detection using Accelerated Object Detection Single Shot MultiBox Deep Learning Models