Abstract:Deep neural networks (DNNs) have made significant achievements in a wide variety of domains. For the deep learning tasks, multiple excellent hardware platforms provide efficient solutions, including graphics processing units (GPUs), central processing units (CPUs), field programmable gate arrays (FPGAs), and application-specific integrated circuit (ASIC). Nonetheless, CPUs outperform other solutions including GPUs in many cases for the inference workload of DNNs with the support of various techniques, such as the high-performance libraries being the basic building blocks for DNNs. Thus, CPUs have been a preferred choice for DNN inference applications, particularly in the low-latency demand scenarios. However, the DNN inference efficiency remains a critical issue, especially when low latency is required under conditions with limited hardware resources, such as embedded systems. At the same time, the hardware features have not been fully exploited for DNNs and there is much room for improvement. To this end, this paper conducts a series of experiments to make a thorough study for the inference workload of prominent state-of-the-art DNN architectures on a single-instruction-multiple-data (SIMD) CPU platform, as well as with widely applicable scopes for multiple hardware platforms. The study goes into depth in DNNs: the CPU kernel-instruction level performance characteristics of DNNs including branches, branch prediction misses, cache misses, etc, and the underlying convolutional computing mechanism at the SIMD level; The thorough layer-wise time consumption details with potential time-cost bottlenecks; And the exhaustive dynamic activation sparsity with exact details on the redundancy of DNNs. The research provides researchers with comprehensive and insightful details, as well as crucial target areas for optimising and improving the efficiency of DNNs at both the hardware and software levels.

Anatomizing Deep Learning Inference in Web Browsers

Close the Gap Between Deep Learning and Mobile Intelligence by Incorporating Training in the Loop

Explore Training of Deep Convolutional Neural Networks on Battery-powered Mobile Devices: Design and Application

Deep Learning on Mobile and Embedded Devices: State-of-the-art, Challenges, and Future Directions

Moving Deep Learning into Web Browser: How Far Can We Go?

Empowering In-Browser Deep Learning Inference on Edge Through Just-In-Time Kernel Optimization.

Empowering In-Browser Deep Learning Inference on Edge Devices with Just-in-Time Kernel Optimizations

Accurate Deep Learning Inference Latency Prediction over Dynamic Running Mobile Devices

An architecture-level analysis on deep learning models for low-impact computations

Parallelizing DNN Inference in Mobile Web Browsers on Heterogeneous Hardware

Cloud-based or On-device: An Empirical Study of Mobile Deep Inference

Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls

Toward Efficient Execution of Mainstream Deep Learning Frameworks on Mobile Devices: Architectural Implications

Deep Learning Inference in Facebook Data Centers: Characterization, Performance Optimizations and Hardware Implications

Benchmarking of DL Libraries and Models on Mobile Devices

Efficient Architecture Paradigm for Deep Learning Inference As a Service.

Automated Customization of On-Device Inference for Quality-of-Experience Enhancement

Woodpecker-DL: Accelerating Deep Neural Networks via Hardware-Aware Multifaceted Optimizations

Profiling and optimizing deep learning inference on mobile GPUs.

Exploring the Boundaries of On-Device Inference: When Tiny Falls Short, Go Hierarchical

Characterizing the Deep Neural Networks Inference Performance of Mobile Applications