Abstract:Modern deep learning applications urge to push the model inference taking place at the edge devices for multiple reasons such as achieving shorter latency, relieving the burden of the network connecting to the cloud, and protecting user privacy. The Convolutional Neural Network (CNN) is one of the most widely used model family in the applications. Given the high computational complexity of the CNN models, it is favorable to execute them on the integrated GPUs at the edge devices, which are ubiquitous and have more power and better energy efficiency than the accompanying CPUs. However, programming on integrated GPUs efficiently is challenging due to the variety of their architectures and programming interfaces. This paper proposes an end-to-end solution to execute CNN model inference on the integrated GPUs at the edge, which uses a unified IR to represent and optimize vision-specific operators on integrated GPUs from multiple vendors, as well as leverages machine learning-based scheduling search schemes to optimize computationally-intensive operators like convolution. Our solution even provides a fallback mechanism for operators not suitable or convenient to run on GPUs. The evaluation results suggest that compared to state-of-the-art solutions backed up by the vendor-provided high-performance libraries on Intel Graphics, ARM Mali GPU, and Nvidia integrated Maxwell GPU, our solution achieves similar, or even better (up to 1.62), performance on a number of popular image classification and object detection models. In addition, our solution has a wider model coverage and is more flexible to embrace new models. Our solution has been adopted in production services in AWS and is open-sourced.

ILP-M Conv: Optimize Convolution Algorithm for Single-Image Convolution Neural Network Inference on Mobile GPUs

Optical Convolution Based Computational Method for Low-Power Image Processing

High Performance Depthwise and Pointwise Convolutions on Mobile Devices

Optimizing GPU Memory Transactions for Convolution Operations

MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications

An Efficient Image to Column Algorithm for Convolutional Neural Networks.

Profiling and optimizing deep learning inference on mobile GPUs.

Quantized Convolutional Neural Networks for Mobile Devices

Tensor Layout Optimization of Convolution for Inference on Digital Signal Processor

High performance ultra-low-precision convolutions on mobile devices

Performance Analysis of GPU-Based Convolutional Neural Networks

ABM-SpConv-SIMD: Accelerating Convolutional Neural Network Inference for Industrial IoT Applications on Edge Devices

Automatic Generation of High-Performance Convolution Kernels on ARM CPUs for Deep Learning

On-Device Neural Net Inference with Mobile GPUs

High Performance Implementation of 3D Convolutional Neural Networks on a GPU

A Unified Optimization Approach for CNN Model Inference on Integrated GPUs

ZNNi - Maximizing the Inference Throughput of 3D Convolutional Networks on Multi-Core CPUs and GPUs

GPNPU: Enabling Efficient Hardware-Based Direct Convolution with Multi-Precision Support in GPU Tensor Cores

Overparametrization of HyperNetworks at Fixed FLOP-Count Enables Fast Neural Image Enhancement

ParallelFusion

ConvReLU++: Reference-based Lossless Acceleration of Conv-ReLU Operations on Mobile CPU