Abstract:Over the last years, deep neural networks (DNNs) are becoming more powerful and have risen in popularity, especially in mobile computing. Applications running on edge AI devices such as smartphones would potentially benefit from the new opportunities enabled by deep learning techniques. However, DNNs are by nature computationally and memory intensive, making them challenging to deploy on mobile devices. Binary neural networks (BNNs) have been considered as a promising solution that can significantly reduce the memory and computational requirements of DNNs while still offering similar capabilities of full precision DNN models. Currently, existing GPU-accelerated implementations of BNNs are only tailored for desktop platforms. Due to architecture differences, mere porting of such implementations to mobile devices yields suboptimal performance or is impossible in some cases. Therefore, there has still been a missing piece in the literature for GPU-accelerated implementations of BNNs on mobile devices. In this paper, we propose PhoneBit, a GPU-accelerated BNN inference engine for mobile devices that fully exploits the computing power of BNNs on mobile GPUs. PhoneBit provides a set of operator-level optimizations including locality-friendly data layout, bit packing with vectorization and layers integration for efficient binary convolution. We also provide a detailed implementation and parallelization optimization for PhoneBit to optimally utilize the memory bandwidth and computing power of mobile GPUs. Our experiment results show that PhoneBit can achieve significant speedup and energy efficiency compared with state-of-the-art frameworks for mobile devices. The PhoneBit open source library is available for download at <a href="https://code.ihub.org.cn/projects/915/repository/PhoneBit">https://code.ihub.org.cn/projects/915/repository/PhoneBit</a>.

Parallelizing DNN Inference in Mobile Web Browsers on Heterogeneous Hardware

MOC: Multi-Objective Mobile CPU-GPU Co-Optimization for Power-Efficient DNN Inference

Extendable Multi-Device Collaborative Pipeline Parallel Inference in the Edge-Cloud Scenario

WebInf: Accelerating WebGPU-based In-browser DNN Inference Via Adaptive Model Partitioning

Thermal-Aware On-Device Inference Using Single-Layer Parallelization with Heterogeneous Processors

AdaKnife: Flexible DNN Offloading for Inference Acceleration on Heterogeneous Mobile Devices

Pipeline Parallelism for Inference on Heterogeneous Edge Computing

Deep Learning Inference on Heterogeneous Mobile Processors: Potentials and Pitfalls

Empowering In-Browser Deep Learning Inference on Edge Devices with Just-in-Time Kernel Optimizations

Hastening Stream Offloading of Inference Via Multi-Exit DNNs in Mobile Edge Computing

PipePar: A Pipelined Hybrid Parallel Approach for Accelerating Distributed DNN Training

Empowering In-Browser Deep Learning Inference on Edge Through Just-In-Time Kernel Optimization.

AdaOper: Energy-efficient and Responsive Concurrent DNN Inference on Mobile Devices

Adaptive Partitioning and Efficient Scheduling for Distributed DNN Training in Heterogeneous IoT Environment

Automatic Pipeline Parallelism: A Parallel Inference Framework for Deep Learning Applications in 6G Mobile Communication Systems

ParallelFusion

Accelerating DNN Inference with Heterogeneous Multi-DPU Engines

Hybrid-Parallel: Achieving High Performance and Energy Efficient Distributed Inference on Robots

An efficient GPU-accelerated inference engine for binary neural network on mobile phones

MNN: A Universal and Efficient Inference Engine

An Adaptive DNN Inference Acceleration Framework with End–edge–cloud Collaborative Computing