Abstract:Deep neural networks (DNNs) have made significant achievements in a wide variety of domains. For the deep learning tasks, multiple excellent hardware platforms provide efficient solutions, including graphics processing units (GPUs), central processing units (CPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuit (ASIC). Nonetheless, CPUs outperform other solutions including GPUs in many cases for the inference workload of DNNs with the support of various techniques, such as the high-performance libraries being the basic building blocks for DNNs. Thus, CPUs have been a preferred choice for DNN inference applications, particularly in the low-latency demand scenarios. However, the DNN inference efficiency remains a critical issue, especially when low latency is required under conditions with limited hardware resources, such as embedded systems. At the same time, the hardware features have not been fully exploited for DNNs and there is much room for improvement. To this end, this paper conducts a series of experiments to make a thorough study for the inference workload of prominent state-of-the-art DNN architectures on a single-instruction-multiple-data (SIMD) CPU platform, as well as with widely applicable scopes for multiple hardware platforms. The study goes into depth in DNNs: the CPU kernel-instruction level performance characteristics of DNNs including branches, branch prediction misses, cache misses, etc, and the underlying convolutional computing mechanism at the SIMD level; The thorough layer-wise time consumption details with potential time-cost bottlenecks; And the exhaustive dynamic activation sparsity with exact details on the redundancy of DNNs. The research provides researchers with comprehensive and insightful details, as well as crucial target areas for optimising and improving the efficiency of DNNs at both the hardware and software levels.

Battle on Edge - Comparison of Convolutional Neural Networks Inference Speed on Two Various Hardware Platforms

Edge Devices Inference Performance Comparison

Benchmarking Edge AI Platforms for High-Performance ML Inference

ABM-SpConv-SIMD: Accelerating Convolutional Neural Network Inference for Industrial IoT Applications on Edge Devices

DistrEdge: Speeding up Convolutional Neural Network Inference on Distributed Edge Devices

An architecture-level analysis on deep learning models for low-impact computations

Exploring Deep Neural Networks on Edge TPU

Evolution of Convolutional Neural Network (CNN): Compute vs Memory bandwidth for Edge AI

Characterizing Resource Heterogeneity in Edge Devices for Deep Learning Inferences

Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning

pCAMP: Performance Comparison of Machine Learning Packages on the Edges

EmBench: Quantifying Performance Variations of Deep Neural Networks across Modern Commodity Devices

Learning on Hardware: A Tutorial on Neural Network Accelerators and Co-Processors

Speeding up Machine Learning Inference on Edge Devices by Improving Memory Access Patterns using Coroutines

Evaluation of Convolution Primitives for Embedded Neural Networks on 32-bit Microcontrollers

Efficient Computer Vision on Edge Devices with Pipeline-Parallel Hierarchical Neural Networks

Optimized CNN Architectures Benchmarking in Hardware-Constrained Edge Devices in IoT Environments

Research on Convolutional Neural Network Inference Acceleration and Performance Optimization for Edge Intelligence

DNN is not all you need: Parallelizing Non-Neural ML Algorithms on Ultra-Low-Power IoT Processors

Empowering In-Browser Deep Learning Inference on Edge Devices with Just-in-Time Kernel Optimizations

Benchmarking Edge Computing Devices for Grape Bunches and Trunks Detection using Accelerated Object Detection Single Shot MultiBox Deep Learning Models