Abstract:Edge devices are seeing tremendous growth in sensing and computational capabilities. Running state-of-the-art deep neural network (NN) based data processing on multi-core CPU processors, embedded Graphics Processing Units (GPU), Tensor Processing Units (TPU), Neural Processing Units (NPU), Deep Learning Accelerators (DLA) etc., edge devices are now able to handle heavy data computations with limited or without cloud connectivity. In addition to hardware resources, software frameworks that optimize a trained neural network (NN) model through weight clustering and pruning, weight and input-output quantization to fewer bits, fusing NN layers etc., for more efficient execution of NN inferences on edge platforms, play an important role in making machine learning at the edge (namely EdgeML) a reality. This paper is a first effort in characterizing these software frameworks for DNN inference optimizations on edge devices, especially edge GPUs which are now ubiquitously used in all embedded deep learning systems. The interactions between software optimizations and the underlying GPU hardware is carefully examined. As most NN optimization engines are proprietary softwares with undocumented internal details in the public domain, our empirical analysis on real embedded GPU platforms using a variety of widely used DNNs, provide various interesting findings. We observe tremendous performance gain and non-negligible accuracy gain from the software optimizations, but also find highly unexpected non-deterministic behaviors such as different outputs on same inputs or increased execution latency for same NN model on more powerful hardware platforms. Application developers using these proprietary software optimization engines, would benefit from our analysis and the discussed implications of our findings, with examples from real applications like intelligent traffic intersection control and Advanced Driving Assistance Systems (ADAS). There are important implications of our findings on performance modeling and prediction research too, that focus on micro-architecture modeling based application performance prediction, but should now additionally consider optimization engines that this paper examines.

On Hardware-Aware Design and Optimization of Edge Intelligence

TinyEdge: Enabling Rapid Edge System Customization for IoT Applications

Efficient Hardware Acceleration Techniques for Deep Learning on Edge Devices: A Comprehensive Performance Analysis

Research on Convolutional Neural Network Inference Acceleration and Performance Optimization for Edge Intelligence

Enabling Design Methodologies and Future Trends for Edge AI: Specialization and Co-design

Circuit and System Technologies for Energy-Efficient Edge Robotics

Automated and Holistic Co-design of Neural Networks and ASICs for Enabling In-Pixel Intelligence

A Survey on Hardware Accelerator Design of Deep Learning for Edge Devices

Bringing AI To Edge: From Deep Learning's Perspective

Design Automation for Efficient Deep Learning Computing

Hardware-Aware Graph Neural Network Automated Design for Edge Computing Platforms

AI on the Edge: Rethinking AI-based IoT Applications Using Specialized Edge Architectures

HAO: Hardware-aware neural Architecture Optimization for Efficient Inference

A Survey on Optimization Techniques for Edge Artificial Intelligence (AI)

Heterogeneous Computing for Edge AI

Custom Hardware Architectures for Deep Learning on Portable Devices: A Review

Demystifying TensorRT: Characterizing Neural Network Inference Engine on Nvidia Edge Devices

A Hardware Co-design Workflow for Scientific Instruments at the Edge

3U-EdgeAI: Ultra-Low Memory Training, Ultra-Low BitwidthQuantization, and Ultra-Low Latency Acceleration

Edge Intelligence: Paving the Last Mile of Artificial Intelligence with Edge Computing