Abstract:Deep neural networks (DNNs) have made significant achievements in a wide variety of domains. For the deep learning tasks, multiple excellent hardware platforms provide efficient solutions, including graphics processing units (GPUs), central processing units (CPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuit (ASIC). Nonetheless, CPUs outperform other solutions including GPUs in many cases for the inference workload of DNNs with the support of various techniques, such as the high-performance libraries being the basic building blocks for DNNs. Thus, CPUs have been a preferred choice for DNN inference applications, particularly in the low-latency demand scenarios. However, the DNN inference efficiency remains a critical issue, especially when low latency is required under conditions with limited hardware resources, such as embedded systems. At the same time, the hardware features have not been fully exploited for DNNs and there is much room for improvement. To this end, this paper conducts a series of experiments to make a thorough study for the inference workload of prominent state-of-the-art DNN architectures on a single-instruction-multiple-data (SIMD) CPU platform, as well as with widely applicable scopes for multiple hardware platforms. The study goes into depth in DNNs: the CPU kernel-instruction level performance characteristics of DNNs including branches, branch prediction misses, cache misses, etc, and the underlying convolutional computing mechanism at the SIMD level; The thorough layer-wise time consumption details with potential time-cost bottlenecks; And the exhaustive dynamic activation sparsity with exact details on the redundancy of DNNs. The research provides researchers with comprehensive and insightful details, as well as crucial target areas for optimising and improving the efficiency of DNNs at both the hardware and software levels.

PRoof: A Comprehensive Hierarchical Profiling Framework for Deep Neural Networks with Roofline Analysis

All-in-One: A Highly Representative DNN Pruning Framework for Edge Devices with Dynamic Power Management

dPRO: A Generic Profiling and Optimization System for Expediting Distributed DNN Training

Hierarchical Roofline analysis for GPUs: Accelerating performance optimization for the NERSC‐9 Perlmutter system

Hierarchical Roofline Analysis: How to Collect Data using Performance Tools on Intel CPUs and NVIDIA GPUs

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

An Optimization Toolchain Design Of Deep Learning Deployment Based On Heterogeneous Computing Platform

Daydream: Accurately Estimating the Efficacy of Optimizations for DNN Training

A Performance Analysis Framework for Exploiting GPU Microarchitectural Capability.

DeepContext: A Context-aware, Cross-platform, and Cross-framework Tool for Performance Profiling and Analysis of Deep Learning Workloads

Applying the Roofline model for Deep Learning performance optimizations

An architecture-level analysis on deep learning models for low-impact computations

Towards Ultra-High Performance and Energy Efficiency of Deep Learning Systems: An Algorithm-Hardware Co-Optimization Framework

Towards Real-Time DNN Inference on Mobile Platforms with Model Pruning and Compiler Optimization

Proteus: Simulating the Performance of Distributed DNN Training

The Sparsity Roofline: Understanding the Hardware Limits of Sparse Neural Networks

DeepProf: Performance Analysis for Deep Learning Applications via Mining GPU Execution Patterns

A DNN Optimization Framework with Unlabeled Data for Efficient and Accurate Reconfigurable Hardware Inference

Optimising the Performance of Convolutional Neural Networks across Computing Systems using Transfer Learning

BitXpro: Regularity-Aware Hardware Runtime Pruning for Deep Neural Networks

A Heterogeneous Full-stack AI Platform for Performance Monitoring and Hardware-specific Optimizations