Abstract:Edge devices are seeing tremendous growth in sensing and computational capabilities. Running state-of-the-art deep neural network (NN) based data processing on multi-core CPU processors, embedded Graphics Processing Units (GPU), Tensor Processing Units (TPU), Neural Processing Units (NPU), Deep Learning Accelerators (DLA) etc., edge devices are now able to handle heavy data computations with limited or without cloud connectivity. In addition to hardware resources, software frameworks that optimize a trained neural network (NN) model through weight clustering and pruning, weight and input-output quantization to fewer bits, fusing NN layers etc., for more efficient execution of NN inferences on edge platforms, play an important role in making machine learning at the edge (namely EdgeML) a reality. This paper is a first effort in characterizing these software frameworks for DNN inference optimizations on edge devices, especially edge GPUs which are now ubiquitously used in all embedded deep learning systems. The interactions between software optimizations and the underlying GPU hardware is carefully examined. As most NN optimization engines are proprietary softwares with undocumented internal details in the public domain, our empirical analysis on real embedded GPU platforms using a variety of widely used DNNs, provide various interesting findings. We observe tremendous performance gain and non-negligible accuracy gain from the software optimizations, but also find highly unexpected non-deterministic behaviors such as different outputs on same inputs or increased execution latency for same NN model on more powerful hardware platforms. Application developers using these proprietary software optimization engines, would benefit from our analysis and the discussed implications of our findings, with examples from real applications like intelligent traffic intersection control and Advanced Driving Assistance Systems (ADAS). There are important implications of our findings on performance modeling and prediction research too, that focus on micro-architecture modeling based application performance prediction, but should now additionally consider optimization engines that this paper examines.

Performance Analysis and Optimization Opportunities for NVIDIA Automotive GPUs

An Evaluation Of The Nvidia Tx1 For Supporting Real-Time Computer-Vision Workloads

Performance/power assessment of CNN packages on embedded automotive platforms

Intelligent Orchestration of ADAS Pipelines on Next Generation Automotive Platforms

Optimizing Monocular Driving Assistance for Real-Time Processing on Jetson AGX Xavier

High-Integrity GPU Designs for Critical Real-Time Automotive Systems

A quantitative performance analysis model for GPU architectures

TensorRT-based Framework and Optimization Methodology for Deep Learning Inference on Jetson Boards

Data-Driven Analysis to Understand GPU Hardware Resource Usage of Optimizations

Demystifying TensorRT: Characterizing Neural Network Inference Engine on Nvidia Edge Devices

Efficient Computing Platform Design for Autonomous Driving Systems

Enabling Level-4 Autonomous Driving on a Single $1k Off-the-Shelf Card

Evaluation of Thermal Imaging on Embedded GPU Platforms for Application in Vehicular Assistance Systems

A Performance Analysis Framework for Exploiting GPU Microarchitectural Capability.

Performance Tuning for GPU-Embedded Systems: Machine-Learning-based and Analytical Model-driven Tuning Methodologies

ProAI: An Efficient Embedded AI Hardware for Automotive Applications -- a Benchmark Study

Performance Implications of Multi-Chiplet Neural Processing Units on Autonomous Driving Perception

Multi-Tenant Virtual GPUs for Optimising Performance of a Financial Risk Application

Understanding the GPU Microarchitecture to Achieve Bare-Metal Performance Tuning

An adaptive GPU performance and power model